diff --git a/.claude/commands/audit-docs.md b/.claude/commands/audit-docs.md new file mode 100644 index 0000000..d3c8297 --- /dev/null +++ b/.claude/commands/audit-docs.md @@ -0,0 +1,71 @@ +--- +description: Audit the prose docs against the actual code and fix any drift, treating every doc claim as untrusted until verified against source +--- + +Audit xray's prose documentation against the current code and bring it back in +sync. The governing rule is [`.claude/rules/docs-freshness.md`](../rules/docs-freshness.md) +— read it first for the source → doc map. + +**Premise: every doc claim is untrusted until verified against source.** A +markdown table, a docstring, a code example — all are claims about the code, not +the code itself. Do not trust a doc because it reads confidently. Open the +source and confirm. + +## Scope + +`$ARGUMENTS` may narrow the audit to one area (e.g. `sdk`, `wire-contract`, +`schema`). If empty, audit the full set: + +- `docs/` — `index.md`, `architecture.md`, `integrate.md`, `sdk-python.md`, `wire-contract.md` +- `README.md`, `CONTRIBUTING.md`, `CLAUDE.md` +- `sdk/python/README.md`, `sdk/python/.claude/rules/typed-boundaries.md` + +If a commit range is given (e.g. `since main`), use `git diff` to find which +source files moved and audit only the docs the map couples them to — a targeted +pre-PR pass. + +## Method + +1. **Walk the source → doc map** in the freshness rule. For each doc in scope, + list its concrete claims: exported names, signatures, enum values, table + counts, env var names + defaults, API routes, limits, status codes, file + paths, code examples. +2. **Verify each claim against source** — Read the file, Grep the symbol. Pick + the highest-signal ground truth: + - SDK public surface → `sdk/python/src/xray/__init__.py` (`__all__`) + the + dataclass / function defs in `conversation.py`, `config.py`, `errors.py`, + `instrument.py`, `orchestrator.py`, `runtime/*`. + - Storage / enums → `src/server/store/schema.ts` (`lifecycle_state`, + `analysis_step`, `failure_reason`, table list). + - OTLP vocabularies / extraction → `src/server/otlp/vocabularies/*`. + - Limits + error shapes → `src/server/otlp/otlp.types.ts`, `otlp.router.ts`. + - Env vars → `src/server/env/env.ts`. +3. **Stale-name sweep.** Grep the doc set for names the code no longer exports + (the highest-confidence drift). Judge each hit — a legitimate "renamed from + X" migration note is fine; a live reference to a dead symbol is not. +4. **Check cross-links.** Every relative markdown link and `github.com/.../blob` + link must resolve. Remember `docs/` is served as a GitHub Pages site under + `baseurl: /xray` — links that escape the `docs/` root must use the full blob + URL, not `../`. +5. **Honesty pass** ([`honesty.md`](../rules/honesty.md)). Flag any claim that + says *more* than the code guarantees, not just outright-wrong ones. Over-claims + are the subtle drift. + +## Fix + +Apply the corrections directly. For each fix, the doc must now match a specific +line of source you can point to. If a claim can't be verified, remove or soften +it rather than leaving it authoritative. + +## Report + +End with a short summary grouped by severity: + +- **Wrong** — a flat-out false claim a reader would act on (fixed). +- **Over-claim** — says more than the code guarantees (fixed/softened). +- **Stale link / name** — dead reference (fixed). +- **Checked, fine** — a claim you verified that turned out correct (list briefly, + so the audit's coverage is legible — per [`honesty.md`](../rules/honesty.md) §3, + don't claim you audited what you didn't). + +Do not commit or push unless asked — leave the working tree for review. diff --git a/.claude/rules/docs-freshness.md b/.claude/rules/docs-freshness.md new file mode 100644 index 0000000..17fd5be --- /dev/null +++ b/.claude/rules/docs-freshness.md @@ -0,0 +1,39 @@ +# Docs track the code — audit them when the surface moves + +The prose docs in this repo (`docs/`, the root `README.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `sdk/python/README.md`) are **untrusted by default**: a comment, a docstring, or a markdown table is a claim about the code, not the code itself, and it drifts silently the moment the thing it describes changes. There is **no CI gate** for this on purpose — AI-in-CI is too expensive for a self-hosted single-image project, and a deterministic denylist of stale strings rots faster than the docs it guards. So freshness is a *process* obligation, enforced by this rule plus the [`/audit-docs`](../commands/audit-docs.md) command, not by a check. + +The failure mode this prevents: someone renames `LiveKitDriver` → `LiveKitRuntime` in the SDK, the rename compiles and the tests pass, but six markdown files still say `LiveKitDriver` — and a reader who copies the example gets an `ImportError`. That was a real, whole-PR cleanup (the docs rewrite that introduced this rule). The fix is cheap *at the point of change* (you're already in the code) and expensive later (a full re-audit). + +--- + +## 1 · The obligation + +When you change a **public surface** — anything a doc describes — re-check the docs it maps to **in the same change**, the same "fix in passing" reflex as [`pattern-matching.md`](./pattern-matching.md) §4 and [`comments.md`](./comments.md) §4. You do not need to rewrite a doc you didn't invalidate; you need to confirm the ones you *did* touch still tell the truth. + +Before opening a PR that touches any row's left column below, run [`/audit-docs`](../commands/audit-docs.md) (or do the equivalent by hand). It's advisory, not gated — but the audit is cheap and the drift is embarrassing in a public repo. + +## 2 · The source → doc map + +Coarse on purpose (directory / file → doc), so it survives refactors that a line-level map wouldn't. If you add a doc or a new public surface, add the edge here. + +| If you change… | Re-check… | +|-------------------------------------------------------------|---------------------------------------------------------------------------| +| `sdk/python/src/xray/__init__.py` (`__all__`), `conversation.py`, `config.py`, `errors.py`, `instrument.py`, `orchestrator.py` (public signatures) | `docs/sdk-python.md` (API table + signatures), `sdk/python/README.md` (quickstart) | +| `sdk/python/src/xray/runtime/*` (runtime classes / ABC / protocols) | `docs/sdk-python.md` (Runtimes), `docs/integrate.md` (example imports) | +| `src/server/store/schema.ts` (tables, `lifecycle_state` / `analysis_step` / `failure_reason` enums) | `docs/architecture.md` (storage ERD + table list), `docs/integrate.md`, `CLAUDE.md` (Storage ¶) | +| `src/server/otlp/vocabularies/*` | `docs/wire-contract.md` (vocabularies), `docs/integrate.md` (vocab section) | +| `src/server/otlp/otlp.types.ts` (size / span caps, error shapes) | `docs/wire-contract.md` (Limits + status codes) | +| `src/server/env/env.ts` (`XRAY_*` env vars + defaults) | `docs/sdk-python.md`, `docs/architecture.md`, `README.md`, `CONTRIBUTING.md` | +| Control-plane routes (`src/server/**/.router.ts`) | `docs/architecture.md` (control-plane list). The OpenAPI at `/docs` self-syncs from `describeRoute` — only the *narrative* needs a human. | + +## 3 · Honesty bar (cross-link) + +A doc must not claim more than the code guarantees — see [`honesty.md`](./honesty.md). "The SDK synthesizes TTS" when TTS moved server-side, or "X is supported" when the code does less, is an honesty failure, not a typo. The `/audit-docs` method is built on this: every doc claim is verified *against source*, and an unverifiable claim is removed or softened, never left to look authoritative. + +--- + +## What's NOT a rule here + +- **"Every code change needs a doc change."** No — most don't (internal refactors, test additions, bug fixes that already match the docs). The obligation fires only when you move a surface a doc *describes*. +- **"Add a CI check / pre-commit grep for stale strings."** Deliberately rejected: maintenance cost + an incomplete denylist gives false confidence. Stale-name grepping lives *inside* `/audit-docs`, where a human judges each hit, not as a standalone gate. +- **"Keep docs exhaustive."** Out of scope. This rule is about *accuracy* of what's documented, not *coverage*. diff --git a/CLAUDE.md b/CLAUDE.md index 8056ed5..a6eaba0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,23 +6,23 @@ **Voice is the primary investment.** Per-turn audio playback, barge-in indicators, per-stage STT/TTS latency, full-replay mixdown — first-class, not afterthoughts. -**Storage.** Conversations, Replays, server-derived `replay_turns` + `speech_segments` (from VAD), per-turn Whisper transcripts (`turn_transcripts`), per-turn timing metrics (`replay_metrics`), per-assertion + per-judge outcomes (`assertion_results`, `judge_results`), the per-replay verdict row (`replay_evaluations`), recognized OTLP spans, and tool-call / model-usage rows live in a single SQLite file at `/data/xray.db` (mounted volume on the container). bunqueue (the embedded job queue) owns a separate `/data/bunqueue.db` file in the same volume — acknowledged tradeoff vs the strict "one file" reading of the single-image rule (single volume, two files, no second process). Both use single-writer `bun:sqlite`, no network driver. Why SQLite is the right choice here is the topic of [`.claude/rules/single-image-distribution.md`](./.claude/rules/single-image-distribution.md). +**Storage.** Conversations, Replays, server-derived `replay_turns` + `speech_segments` (from VAD), per-turn transcripts (`turn_transcripts`), per-turn timing metrics (`replay_metrics`), per-assertion + per-judge outcomes (`assertion_results`, `judge_results`), the per-replay verdict row (`replay_evaluations`), recognized OTLP spans, tool-call / model-usage rows, and the server-side TTS cache (`tts_synth_cache`) live in a single SQLite file at `/data/xray.db` (mounted volume on the container). bunqueue (the embedded job queue) owns a separate `/data/bunqueue.db` file in the same volume — acknowledged tradeoff vs the strict "one file" reading of the single-image rule (single volume, two files, no second process). Both use single-writer `bun:sqlite`, no network driver. Why SQLite is the right choice here is the topic of [`.claude/rules/single-image-distribution.md`](./.claude/rules/single-image-distribution.md). ## The two paths data takes into xray xray has exactly two write surfaces; both are documented and Valibot-validated at the boundary. 1. **Control plane (the SDK calls these directly).** - - `POST /v1/conversations` — idempotent upsert of the Conversation spec (turns + per-turn assertions + conversation-level judges) keyed by its content hash. The server computes the hash; the SDK never hashes anything. Same canonical JSON in → same hash → row upsert (last-write-wins on `name`). - - `POST /v1/replays` — eager Replay-row creation (`lifecycle_state='pending'`). Returns `replay_id` so the SDK can propagate it (LiveKit room metadata → OTEL baggage) BEFORE the dev's agent emits its first span. + - `POST /v1/conversations` — idempotent upsert of the Conversation spec, sent as **multipart/form-data**: a JSON `spec` part (`{name, turns, judges?, live?}`) plus one file part per `RecordedAudio` turn. Keyed by content hash; the server computes the hash (including the sha256 of any audio bytes), the SDK never hashes anything. Same canonical spec in → same hash → row upsert (last-write-wins on `name`). + - `POST /v1/replays` — eager Replay-row creation (`lifecycle_state='pending'`). Returns the Replay row (`ReplayDetailResponse`; its `id` is the replay id) so the SDK can propagate the replay context (the joining participant's JWT `xray` attribute → OTEL baggage; no room/participant metadata) BEFORE the dev's agent emits its first span. - `POST /v1/replays/:id/audio` — driver uploads the 48kHz int16 **stereo WAV** (L = user, R = agent, wall-clock-aligned), plus the `X-Recording-Started-At` header: the wall-clock of audio sample 0, persisted as `replays.recording_started_at` — the sole origin for mapping span timestamps onto the audio timeline. Flips `lifecycle_state` to `recording_uploaded`. - - `POST /v1/replays/:id/analyze` — enqueues the bunqueue `analyze-replay` job. The server transitions to `lifecycle_state='analyzing'` with `analysis_step='vad'`. **Three-stage chain**: `analyze-replay` (VAD per channel + Whisper transcription per turn) → `calculate-metrics` (audio-frame metrics: agent_response_ms, interrupted) → `evaluate-replay` (runs declared assertions + judges, writes `assertion_results` / `judge_results` / `replay_evaluations`, flips lifecycle to `completed`). Each stage enqueues the next on success; any stage's failure stamps `lifecycle_state='failed'` with a stage-specific `failure_reason`. Tool/model spans are attributed to turns at eval/read time by mapping their wall-clock `started_at` onto the audio timeline (origin = `replays.recording_started_at`, see `docs/specs/0001-timeline-clock-alignment.md`) — there is no stored `turn_idx` on those rows and no backfill stage. + - `POST /v1/replays/:id/analyze` — enqueues the bunqueue `analyze-replay` job. The server transitions to `lifecycle_state='analyzing'` with `analysis_step='vad'`. **Three-stage chain**: `analyze-replay` (VAD per channel + per-turn transcription; provider-selectable — OpenAI Whisper / Google Gemini / Mistral Voxtral) → `calculate-metrics` (audio-frame metrics: agent_response_ms, interrupted) → `evaluate-replay` (runs declared assertions + judges, writes `assertion_results` / `judge_results` / `replay_evaluations`, flips lifecycle to `completed`). Each stage enqueues the next on success; any stage's failure stamps `lifecycle_state='failed'` with a stage-specific `failure_reason`. Tool/model spans are attributed to turns at eval/read time by mapping their wall-clock `started_at` onto the audio timeline (origin = `replays.recording_started_at`) — there is no stored `turn_idx` on those rows and no backfill stage. - `GET /v1/replays/:id/events` — SSE stream of `state` / `progress` / `evaluation_complete` / `failed` events. The `evaluation_complete` event carries the full `ReplayResult` payload (verdict + per-assertion + per-judge + per-turn metrics) — the SDK returns immediately without a follow-up GET. - `GET /v1/replays/:id/result` — same `ReplayResult` payload outside the SSE stream for late subscribers / inspector hydration. - `PATCH /v1/replays/:id` — driver-side failures only (`failure_reason='driver_aborted'` / `audio_missing` / `agent_not_joined`). Lifecycle transitions during the analyze chain are server-owned. 2. **OTLP/HTTP receiver (the dev's agent emits spans).** - - `POST /v1/otlp/v1/traces` — OpenTelemetry traces (JSON + protobuf). **Filters, not gates**: routes spans by the `xray.replay.id` resource attribute and runs each through a vocabulary registry (`src/server/otlp/vocabularies/`: `xray.*`, OTel GenAI semconv `gen_ai.*`, Langfuse). Unknown vocabularies are dropped silently; unknown replay ids are dropped silently. Extracted fields land in `tool_calls` and `model_usage` with no stored turn association — turn membership is derived at eval/read time from each row's wall-clock `started_at` against the audio timeline (origin = `replays.recording_started_at`, sent by the driver as the `X-Recording-Started-At` header on the audio upload). Model TTFT, when the agent's instrumentation emits `gen_ai.response.time_to_first_chunk`, lands on `model_usage.ttft_ms`. Every accepted span lands in `spans`. `xray.turn` / `xray.stage.*` are accepted as raw spans only. `xray.assertion` and `xray.judge` are no longer recognized — evaluation runs from the declared `Assertion` / `Judge` catalog. + - `POST /v1/otlp/v1/traces` — OpenTelemetry traces (JSON + protobuf). **Filters, not gates**: routes spans by the `xray.replay.id` attribute (a span-level value overrides the resource-level value) and runs each through a vocabulary registry (`src/server/otlp/vocabularies/`: `xray.*`, OTel GenAI semconv `gen_ai.*`, Langfuse). Unknown vocabularies are dropped silently; unknown replay ids are dropped silently. Extracted fields land in `tool_calls` and `model_usage` with no stored turn association — turn membership is derived at eval/read time from each row's wall-clock `started_at` against the audio timeline (origin = `replays.recording_started_at`, sent by the driver as the `X-Recording-Started-At` header on the audio upload). Model TTFT, when the agent's instrumentation emits `gen_ai.response.time_to_first_chunk`, lands on `model_usage.ttft_ms`. Every accepted span lands in `spans`. `xray.turn` / `xray.stage.*` are accepted as raw spans only. `xray.assertion` and `xray.judge` are no longer recognized — evaluation runs from the declared `Assertion` / `Judge` catalog. The two paths are coupled by trust: the OTLP receiver doesn't create Conversation or Replay rows, ever. The trust boundary is the SDK's POST. The analyze chain adds three "internal" write paths — the embedded bunqueue workers writing `replay_turns` + `speech_segments` + `turn_transcripts` + `replay_metrics` + `assertion_results` + `judge_results` + `replay_evaluations` after reading the uploaded WAV + the declared spec. @@ -42,7 +42,7 @@ The shared `OpenAPIV3.SchemaObject` helper lives in `src/server/core/types.ts` a The Python SDK lives at [`sdk/python/`](./sdk/python). Public surface: - `xray.conversation` — test definitions: `Conversation`, `Turn.user(...)`, `Turn.agent(...)`, declarative `Assertion.contains(...)` / `Assertion.tool_called(...)` / `Assertion.max_latency_ms(...)` / etc., and conversation-level `Judge.text_match(...)`. All assertion / judge variants ship on the wire and run **server-side**. -- `xray.runtime` — pluggable driver ABC; `xray.runtime.livekit.LiveKitDriver` is the v1 implementation. Pipecat / OpenAI Realtime / Gemini Live / raw WebSocket are on the roadmap as new sub-modules. +- `xray.runtime` — pluggable `Runtime` ABC (`xray.runtime.base.Runtime`); `xray.runtime.livekit.LiveKitRuntime` is the v1 implementation, and `xray.runtime.livekit_live.LiveKitLiveRuntime` powers `xray.run_live` (OS-mic sessions). Pipecat / OpenAI Realtime / Gemini Live / raw WebSocket are on the roadmap as new sub-modules. - `xray.attach(ctx)` — async context manager for LiveKit Agents worker entrypoints; reads the JWT's `xray` attribute, installs the OTLP exporter, force-flushes spans on exit. - `xray.run(...)` — orchestrator: POST conversation + replay, drive the driver, upload the stereo WAV, POST `/analyze`, wait for the server's `evaluation_complete` SSE, return `xray.ReplayResult`. Per-assertion / per-judge failures don't raise — `assert result.passed` is the pytest idiom. Driver-side or server-chain failures raise typed `XrayError` / `ReplayEvaluationError`. @@ -69,6 +69,7 @@ Every CI step must be runnable on a developer machine with one command — image @.claude/rules/supply-chain.md @.claude/rules/public-repo.md @.claude/rules/single-image-distribution.md +@.claude/rules/docs-freshness.md ### Python SDK rules (load when touching `sdk/python/`) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5ef3f2b..1893716 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,7 @@ Thanks for considering a contribution. This repo is intentionally small and easy - **pnpm only.** `npm install`, `yarn add`, and `bun install` are blocked by a `preinstall` hook. Use `corepack enable` to pick up the pinned pnpm version. - **Supply chain is non-negotiable.** Read [`.claude/rules/supply-chain.md`](./.claude/rules/supply-chain.md) before adding any dependency. The 7-day cooldown, deny-by-default lifecycle scripts, and SHA-pinned GitHub Actions are not optional. - **This repo is public.** Read [`.claude/rules/public-repo.md`](./.claude/rules/public-repo.md) before touching `.env`, the Dockerfile, or commit messages. A leaked secret must be **rotated**, never rewritten. -- **Conventional commits.** Enforced by commitlint in the `commit-msg` hook. Examples: `feat(adapter): add retell adapter`, `fix(server): handle missing api key`. +- **Conventional commits.** Enforced by commitlint in the `commit-msg` hook. Examples: `feat(server): add Mistral judge provider`, `fix(server): handle missing api key`. - **No `as` casts.** Banned by a custom Biome plugin ([`biome-plugins/no-as-cast.grit`](./biome-plugins/no-as-cast.grit)). The **only** allowed form is `as const` (literal-type widening control — not a cast). For everything else: use `satisfies`, a type guard, or `ts-pattern.match`. ## First-time setup @@ -43,7 +43,7 @@ uv venv uv pip install -e '.[dev]' ``` -`uv` itself is pinned via `.tool-versions` (install: `curl -LsSf https://astral.sh/uv/install.sh | sh` or `brew install uv`). +Install `uv` if you don't have it (`curl -LsSf https://astral.sh/uv/install.sh | sh` or `brew install uv`). CI installs it via a SHA-pinned `setup-uv` action; `.tool-versions` pins Node/Bun/pnpm/Python — not uv. ## Daily loop @@ -52,17 +52,17 @@ pnpm dev # Single Bun container serving SPA + API on :8080 with HMR pnpm typecheck # tsc --noEmit pnpm check # biome check (lint + format) pnpm check:fix # biome check --write -pnpm test # bun test -pnpm test:watch # bun test --watch (the TDD loop) -pnpm test:coverage # bun test --coverage — same gate CI runs -pnpm docker:smoke # build the production image, run it, curl /healthz, kill it +pnpm test # bun test --isolate +pnpm test:watch # bun test --isolate --watch (the TDD loop) +pnpm test:coverage # bun test --isolate --coverage — same gate CI runs +pnpm docker:smoke # build the image, run it, wait for the container HEALTHCHECK (probes /healthz) to go healthy ``` ### TDD Every behavior lands red → green → refactor. The failing test goes in *first*, runs (and fails for the right reason), then the production code makes it green. See [`.claude/rules/tdd.md`](./.claude/rules/tdd.md). The CI `test` workflow runs `pnpm test:coverage` and fails the build if coverage drops below the thresholds in `bunfig.toml`. -`pnpm docker:smoke` is the **single most important** local check — it is exactly what CI runs before publishing. If it passes locally, it passes in CI. +`pnpm docker:smoke` is the **single most important** local check — it builds the production image, runs it, and waits for the container's `HEALTHCHECK` (which probes `/healthz`) to report healthy. CI runs the same script in the `smoke` job of `build.yml` on every PR and push. If it passes locally, it passes in CI. ## Parallel worktrees @@ -108,19 +108,18 @@ git branch -D feat/foo # name shown in .worktree-info ## Code layout -- **Vertical slices, not technical layers.** `src/adapters/elevenlabs/`, `src/graph/`, `src/inspector/` — each folder owns its own components, hooks, types, helpers. No top-level `components/`, `hooks/`, `services/`, `utils/` god-folders. +- **Vertical slices, not technical layers.** `src/server/conversations/`, `src/server/otlp/vocabularies/`, `src/client/inspector/` — each folder owns its own router/service/types/tests (client slices own their components/hooks). No top-level `components/`, `hooks/`, `services/`, `utils/` god-folders. - **Tests next to the source.** `foo.ts` and `foo.test.ts` live in the same folder. No `tests/` mirror tree, no `__tests__/`. See [`.claude/rules/code-layout.md`](./.claude/rules/code-layout.md) for the full rationale. -## Adding a provider adapter +## Extending xray -The core app is provider-agnostic. Adding a new provider is one file plus tests; no UI changes. +There is no server-side "adapter" system — the alpha rewrite removed it. The current extension points are: -1. Read [`src/adapters/types.ts`](./src/adapters/types.ts) — the `VoiceAgentAdapter` interface and the provider-agnostic types (`Agent`, `Workflow`, `Conversation`, `Turn`). -2. Create the slice: `src/adapters//adapter.ts` implementing `VoiceAgentAdapter`, plus any ``-specific helpers / types in the same folder. Tests live next to the file they test (`adapter.test.ts`). -3. Register it in `src/adapters/registry.ts`. -4. Add a row to the supported-providers table in the README. +- **A new transport / runtime (SDK side).** Subclass `Runtime` from [`sdk/python/src/xray/runtime/base.py`](./sdk/python/src/xray/runtime/base.py) and implement its abstract methods; `LiveKitRuntime` is the reference. Pipecat / OpenAI Realtime / raw WebSocket would each be a new module under `sdk/python/src/xray/runtime/`. +- **A new OTLP vocabulary.** Drop one file in [`src/server/otlp/vocabularies/`](./src/server/otlp/vocabularies/) and add one line to `registry.ts`. Each vocabulary is a pure `match(span, resource)` function — test it against synthetic spans with the slice's test-utils. +- **A new transcription / TTS / judge provider.** Add one file in `src/server/transcription/`, `src/server/tts/`, or `src/server/judges/`, then wire it into the selector in [`src/server/providers/providers.ts`](./src/server/providers/providers.ts). Providers are chosen at runtime via `XRAY_TRANSCRIPTION_PROVIDER` / `XRAY_TTS_PROVIDER` / `XRAY_JUDGE_PROVIDER`. -The interface is deliberately small — if your provider needs something the interface can't express, open an issue first. Interface churn affects every adapter. +Tests live next to the file they test. See [`.claude/rules/code-layout.md`](./.claude/rules/code-layout.md). ## Adding a dependency diff --git a/README.md b/README.md index c1f1566..24a47ad 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ docker build -t xray:local . The Python SDK: ```bash -pip install xray-py[livekit] +pip install "xray-py[livekit]" ``` --- @@ -66,8 +66,8 @@ services: my-voice-agent: build: . environment: - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: http://xray:8080/v1/otlp/v1/traces - OTEL_EXPORTER_OTLP_PROTOCOL: http/json + # xray.attach reads this and POSTs OTLP/JSON to ${endpoint}/v1/otlp/v1/traces + XRAY_OTLP_ENDPOINT: http://xray:8080 depends_on: - xray @@ -125,7 +125,7 @@ asyncio.run(main()) ### Wire your agent (one-time) -The dev's agent reads `xray.replay.id` (plus `conversation.id` / `version` / `modality`) from LiveKit room metadata and propagates them as OTEL baggage so every span — `xray.*`, `gen_ai.*`, Langfuse — gets routed to the right Replay. See [`docs/SDK.md`](docs/SDK.md). +The dev's agent reads the replay context (`replay_id`, `conversation_hash`, `modality`) from the joining participant's JWT `xray` attribute (via `participant.attributes`) and propagates it as OTEL baggage so every span — `xray.*`, `gen_ai.*`, Langfuse — gets routed to the right Replay. No room/participant metadata is set. See [`docs/sdk-python.md`](docs/sdk-python.md). --- @@ -150,7 +150,7 @@ One Bun process serves both the SPA and the API. One SQLite file at `/data/xray. │ ▼ ┌─ dev's agent ─────────────────────────────────────────────────────────┐ - │ reads replay.id from room metadata → OTEL baggage │ + │ reads replay context from JWT 'xray' attribute → OTEL baggage │ │ emits xray.* / gen_ai.* / langfuse spans │ └─────────────────────────────────────────────────────────────────────┘ │ OTLP/JSON @@ -178,8 +178,10 @@ One Bun process serves both the SPA and the API. One SQLite file at `/data/xray. ## Documentation -- [`docs/SDK.md`](docs/SDK.md) — Python authoring + runtime + how to propagate baggage from LiveKit room metadata. -- [`docs/WIRE.md`](docs/WIRE.md) — OTLP attribute contract + recognized vocabularies and what fields are extracted from each. +- [`docs/sdk-python.md`](docs/sdk-python.md) — Python SDK reference: authoring Conversations, runtimes, `attach`, `run` / `run_live`, and how the replay context propagates from the JWT `xray` attribute. +- [`docs/wire-contract.md`](docs/wire-contract.md) — OTLP attribute contract + recognized vocabularies and what fields are extracted from each. +- [`docs/architecture.md`](docs/architecture.md) — the three processes, two write paths, and the analyze chain. +- [`docs/integrate.md`](docs/integrate.md) — end-to-end walkthrough: wire your agent and write a test. - `/docs` on your running instance — generated OpenAPI 3.1 reference rendered by Scalar. --- diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 0000000..ce7a16a --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,32 @@ +# GitHub Pages config for the docs/ folder. +# +# Hosting: repo Settings → Pages → "Deploy from a branch" → branch `main`, +# folder `/docs`. GitHub's managed Jekyll build renders this folder server-side +# — there is no workflow file to maintain and nothing is added to the repo's +# pnpm lockfile or CI. +# +# Theme: just-the-docs (sidebar nav + search) is pulled in via remote_theme and +# resolved by GitHub's Pages build, not by this repo's dependencies. To drop the +# external gem entirely, replace the `remote_theme:` line with a built-in theme, +# e.g. `theme: jekyll-theme-cayman` (no nav sidebar, but zero external gems). + +title: xray +description: Open-source, self-hosted replay/eval framework for LiveKit voice agents +remote_theme: just-the-docs/just-the-docs + +# Project-page paths assume the repo is published at github.com/xray-eval/xray. +# If you fork/rename, update both to .github.io and /. +url: https://xray-eval.github.io +baseurl: /xray + +color_scheme: dark +search_enabled: true + +# Render the mermaid diagrams in architecture.md / integrate.md (loaded from a +# CDN at view time by just-the-docs; not a build dependency). +mermaid: + version: "10.9.0" + +aux_links: + "GitHub": + - "https://github.com/xray-eval/xray" diff --git a/docs/architecture.md b/docs/architecture.md index 791eddf..d844986 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,3 +1,9 @@ +--- +layout: default +title: Architecture +nav_order: 2 +--- + # xray architecture This doc is the map for anyone contributing to xray. It explains the @@ -26,7 +32,7 @@ End-user integration instructions live in [`integrate.md`](./integrate.md). - Storage is one **SQLite file** at `/data/xray.db` plus the bunqueue job DB at `/data/bunqueue.db`, plus audio bytes on disk under `XRAY_AUDIO_ROOT`. No external services. No second container. See - [`single-image-distribution.md`](../.claude/rules/single-image-distribution.md) + [`single-image-distribution.md`](https://github.com/xray-eval/xray/blob/main/.claude/rules/single-image-distribution.md) for why this is non-negotiable. - The **inspector SPA** is served by the same Bun process that owns the API — one image, one port, one volume. @@ -40,7 +46,7 @@ flowchart LR subgraph DRV["Driver — test side (Python)"] direction TB D1["xray.run(...)
orchestrator
(POSTs control plane,
installs OTLP pipeline,
attaches replay baggage,
waits via SSE)"] - D2["LiveKitDriver
plays user audio,
captures agent audio + transcripts,
writes wall-clock stereo WAV
(L = user, R = agent)"] + D2["LiveKitRuntime
plays user audio,
captures agent audio + transcripts,
writes wall-clock stereo WAV
(L = user, R = agent)"] D1 --> D2 end @@ -59,8 +65,8 @@ flowchart LR direction TB CTL["Control plane
POST /v1/conversations
POST /v1/replays
POST /v1/replays/:id/audio
POST /v1/replays/:id/analyze
GET /v1/replays/:id/events (SSE)
PATCH /v1/replays/:id
GET /v1/conversations
GET /v1/replays/:id"] OTLP["OTLP receiver
POST /v1/otlp/v1/traces
JSON + protobuf

Vocabulary registry:
xray.* + gen_ai.* + Langfuse
routes by xray.replay.id"] - JOB["bunqueue worker
analyze-replay job:
VAD on each channel,
derive turn boundaries,
write replay_turns +
speech_segments"] - DB[("SQLite
/data/xray.db

conversations, replays,
replay_turns, speech_segments,
tool_calls, model_usage, spans")] + JOB["bunqueue worker
3-stage analyze chain:
analyze-replay (VAD + transcribe)
→ calculate-metrics
→ evaluate-replay"] + DB[("SQLite
/data/xray.db

conversations, replays,
replay_turns, speech_segments, spans,
tool_calls, model_usage, turn_transcripts,
replay_metrics, assertion_results,
judge_results, replay_evaluations, tts_synth_cache")] BQDB[("bunqueue DB
/data/bunqueue.db
(jobs, DLQ)")] AUDIO[("Audio
$XRAY_AUDIO_ROOT/
<replay>/replay.wav")] SPA["Inspector SPA
(React, served via
Bun.serve HTML routes)"] @@ -166,11 +172,13 @@ flowchart TB in order: 1. `POST /v1/conversations` — Valibot-validated upsert keyed by - `hash`. Multipart body: a `spec` JSON part with `name` + `turns`, - plus one named file part per `RecordedAudio` turn keyed by the - turn's declared `upload_key`. The server reads each audio part, - sha256s the bytes, substitutes the hash into the canonical turn, - then hashes the canonical turn JSON to derive `conversation_hash`. + `hash`. Multipart body: a `spec` JSON part with `name` + `turns` + (+ optional `judges` / `live`), plus one named file part per + `RecordedAudio` turn keyed by the turn's declared `upload_key`. The + server reads each audio part, sha256s the bytes, substitutes the hash + into the canonical turn, then hashes the canonical spec JSON + (`{turns, judges}`) to derive `conversation_hash` — so changing a judge + forks a new Conversation. Re-POSTing the same hash with a different `name` updates the row's display label (last-write-wins). The SDK never hashes anything. @@ -232,14 +240,18 @@ eval/read time by mapping `started_at` onto the audio timeline (`audio_offset_ms = started_at − replays.recording_started_at`, the anchor the driver sends via the `X-Recording-Started-At` upload header) and testing the VAD-derived turn window `[turn_start_ms, turn_end_ms)`. -There is no `turn_idx` column on those tables and no backfill stage — -see `docs/specs/0001-timeline-clock-alignment.md` for why -`replays.started_at` (row-creation time) must never be used as the -origin. - -**gen_ai semconv** (`gen-ai-semconv.ts`) — `gen_ai.tool` → `tool_calls`, -`gen_ai.client.operation` → `model_usage`. **Langfuse** vocabulary -(`langfuse.ts`) extracts the same shapes from Langfuse-flavoured GenAI. +There is no `turn_idx` column on those tables and no backfill stage. The +origin is always `replays.recording_started_at` (the audio sample-0 +wall-clock); `replays.started_at` (row-creation time, which precedes the +recording by the room-connect + agent-join latency) must never be used. + +**gen_ai semconv** (`gen-ai-semconv.ts`) — dispatches on +`gen_ai.operation.name`: `execute_tool` → `tool_calls`; `chat` / +`text_completion` → `model_usage` (model TTFT lifted from +`gen_ai.response.time_to_first_chunk`, seconds → ms). **Langfuse** +vocabulary (`langfuse.ts`) extracts the same shapes from Langfuse +observations: `generation` → `model_usage`, `tool` → `tool_calls`. See +[`wire-contract.md`](./wire-contract.md) for the full attribute contract. --- @@ -285,7 +297,7 @@ sequenceDiagram W->>W: compute agent_response_ms
+ interrupted per turn W->>X: insert replay_metrics
analysis_step='evaluate'
enqueue evaluate-replay X->>W: bunqueue enqueue evaluate-replay - W->>W: run each declared Assertion
(pure ts-pattern dispatch)
+ each declared Judge
(OpenAI Chat Completions) + W->>W: run each declared Assertion
(pure ts-pattern dispatch)
+ each declared Judge
(LLM via configured provider:
OpenAI / Google Gemini / Mistral) W->>X: insert assertion_results + judge_results + replay_evaluations
lifecycle_state='completed' X-->>D: SSE 'evaluation_complete' event
(full ReplayResult payload) ``` @@ -320,9 +332,9 @@ erDiagram replays ||--|| replay_evaluations : "replay_id (verdict)" conversations { - text hash PK "SHA-256 of canonical turn JSON (incl. sha256 of RecordedAudio bytes)" + text hash PK "SHA-256 of canonical spec JSON {turns (incl. assertions + sha256 of RecordedAudio bytes), judges}" text name "Free-form display label; last-write-wins on re-POST" - text turns_json "JSON-encoded canonical turn array — the hash input" + text turns_json "canonical JSON of the spec {turns, judges} — the hash input" text created_at text last_run_at "Bumped on every POST /v1/conversations" } @@ -330,8 +342,9 @@ erDiagram text id PK text conversation_hash FK text lifecycle_state "pending | running | recording_uploaded | analyzing | completed | failed" - text analysis_step "vad | turns | null" - text failure_reason "stalled | timeout | explicit_fail | max_attempts_exceeded | worker_lost | upload_failed | driver_aborted | null" + text analysis_step "vad | transcribe | metrics | evaluate | null" + text failure_reason "14 values: driver_aborted | audio_missing | agent_not_joined | upload_failed | missing_credential | transcription_failed | metrics_failed | evaluation_failed | spec_vad_mismatch | stalled | timeout | explicit_fail | max_attempts_exceeded | worker_lost | null" + text recording_started_at "wall-clock (ISO) of audio sample 0 — sole origin for span→turn attribution; null before audio upload" text started_at text finished_at text audio_path "relative path under XRAY_AUDIO_ROOT to the stereo WAV" @@ -401,7 +414,7 @@ flowchart LR Every read endpoint is in `src/server//.router.ts`. The service layer (`.service.ts`) does the actual SQL via Drizzle on `bun:sqlite`. The slice convention is documented in -[`code-layout.md`](../.claude/rules/code-layout.md). +[`code-layout.md`](https://github.com/xray-eval/xray/blob/main/.claude/rules/code-layout.md). --- @@ -411,9 +424,11 @@ Shipped artifact: a Docker image published to GHCR (`ghcr.io/xray-eval/xray`) by CI on tagged releases. Operators run ``` -docker run -v ./data:/data -e XRAY_AUDIO_ROOT=/data/audio ghcr.io/xray-eval/xray +docker run -v ./data:/data ghcr.io/xray-eval/xray ``` +(`XRAY_AUDIO_ROOT` defaults to `/audio`, so it needn't be passed.) + and that is the install. The image carries the Bun process, the pre-built SPA, the SQLite schema (migrated at startup), the bunqueue worker (embedded, same process), and nothing else. No SaaS. No hosted @@ -423,7 +438,7 @@ This single-image promise is load-bearing for several other choices in the codebase (SQLite over Postgres, `bun:sqlite` over a network driver, embedded reads over a separate query service, embedded bunqueue worker over a separate queue process). See -[`single-image-distribution.md`](../.claude/rules/single-image-distribution.md) +[`single-image-distribution.md`](https://github.com/xray-eval/xray/blob/main/.claude/rules/single-image-distribution.md) before proposing any change that would break it. **Two SQLite files in `/data/`.** xray owns `xray.db` (conversations, diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..f381d8d --- /dev/null +++ b/docs/index.md @@ -0,0 +1,48 @@ +--- +layout: default +title: Home +nav_order: 1 +--- + +# xray docs + +**xray** is an open-source, self-hosted replay/eval framework for LiveKit +voice agents. You author a conversation in Python, run it against your agent, +and inspect every turn — audio, transcripts, tool calls, model usage, timings, +and pass/fail verdicts. One Docker image, one SQLite file, one Python SDK. No +accounts, no telemetry, no external database. + +```bash +docker run -v ./data:/data -p 127.0.0.1:8080:8080 ghcr.io/xray-eval/xray:latest +# open http://localhost:8080 · API reference at /docs +pip install "xray-py[livekit]" +``` + +## Where to go + +| Doc | Read it when | +|---|---| +| [Architecture](./architecture.md) | You want the mental model: the three processes, the two write paths, the analyze chain, and storage. | +| [Integrate](./integrate.md) | You have a LiveKit Agents worker and want xray to record + evaluate it. The end-to-end walkthrough. | +| [Python SDK](./sdk-python.md) | The authoritative `xray-py` reference: `Conversation` / `Turn` / `Assertion` / `Judge`, `run` / `run_live`, `attach`, the runtimes. | +| [Wire contract](./wire-contract.md) | You're checking what your agent's OTLP spans must look like to be recognized. | + +The live API reference (OpenAPI 3.1, rendered by Scalar) is at `/docs` on any +running instance. + +## How it fits together + +1. **Author** a `Conversation` — user turns, per-turn `Assertion`s, and an + optional conversation-level `Judge` — in Python. +2. **Run** it with `xray.run(...)`. The SDK joins your LiveKit room as a + user-side participant, plays the user audio, captures the agent's audio + + transcript, and uploads a stereo WAV. +3. Your agent emits **OpenTelemetry spans** during the run; xray's OTLP + receiver routes them by `xray.replay.id` and extracts tool calls + model + usage. +4. The server **analyzes** the recording (VAD → per-turn transcription → + metrics → assertions + judges) and returns a `ReplayResult`. +5. **Inspect** and **compare** runs in the UI. + +> **Alpha.** The wire and SDK API can break between minor versions. Upgrading +> wipes data — delete `/data/xray.db` before starting a new container. diff --git a/docs/integrate.md b/docs/integrate.md index 7d204e0..f9f8bd8 100644 --- a/docs/integrate.md +++ b/docs/integrate.md @@ -1,3 +1,9 @@ +--- +layout: default +title: Integrate +nav_order: 3 +--- + # Integrating xray into an existing LiveKit Agents worker This is the canonical walkthrough. If you've got a LiveKit Agents @@ -69,7 +75,7 @@ span. ## 1. Install the SDK on the agent side ```bash -pip install xray-py[livekit] +pip install "xray-py[livekit]" ``` The `[livekit]` extra pulls in `livekit` + `livekit-api`. Drop it if @@ -132,13 +138,15 @@ Recognized vocabularies (`xray.*`, OTel GenAI `gen_ai.*`, Langfuse `langfuse.*`) get persisted as raw spans AND extracted into structured rows where the vocabulary supports it: -- `gen_ai.tool` → `tool_calls` row. -- `gen_ai.client.operation` (and Langfuse equivalents) → `model_usage` - row. -- `xray.turn` / `xray.stage.*` spans land in the raw `spans` table for - the inspector's timeline. They carry no structured payload — server - evaluation runs from the declared `Assertion` / `Judge` variants, not - from driver-emitted spans. +- `gen_ai.operation.name = execute_tool` → `tool_calls` row. +- `gen_ai.operation.name = chat` / `text_completion` (and the Langfuse + `generation` observation) → `model_usage` row. +- `xray.turn` / `xray.stage.stt` / `xray.stage.tts` spans land in the raw + `spans` table for the inspector's timeline. They carry no structured + payload — server evaluation runs from the declared `Assertion` / + `Judge` variants, not from driver-emitted spans. + +The full attribute contract is in [`wire-contract.md`](./wire-contract.md). Spans from unrecognized vocabularies are dropped silently — that's the "filter, not a gate" design so noisy framework spans don't fill the DB. @@ -156,7 +164,7 @@ import asyncio import xray from xray import Assertion, Judge from xray.conversation import RecordedAudio -from xray.runtime.livekit import LiveKitDriver +from xray.runtime.livekit import LiveKitRuntime async def main(): @@ -187,7 +195,7 @@ async def main(): ), ) - driver = LiveKitDriver( + driver = LiveKitRuntime( url="ws://localhost:7880", api_key="devkey", api_secret="devsecret32charsminimumlengthxyz123", @@ -215,7 +223,7 @@ In pytest the same call is one assertion: ```python async def test_booking_happy_path(): result = await xray.run(conversation=conv, runtime=driver, xray_url=XRAY_URL) - assert result.passed, format_failures(result) + assert result.passed, xray.format_failures(result) ``` `xray.run` is async — wrap in `asyncio.run` for sync test harnesses. @@ -324,7 +332,9 @@ xray ships as a single Docker image. Two SQLite files share the mounted volume: - `/data/xray.db` — conversations, replays, replay_turns, - speech_segments, tool_calls, model_usage, spans. + speech_segments, spans, tool_calls, model_usage, turn_transcripts, + replay_metrics, assertion_results, judge_results, replay_evaluations, + tts_synth_cache (13 tables; see [`architecture.md`](./architecture.md)). - `/data/bunqueue.db` — bunqueue's job queue + DLQ (the `analyze-replay` worker runs embedded in the same Bun process). diff --git a/docs/sdk-python.md b/docs/sdk-python.md new file mode 100644 index 0000000..d5fd4ee --- /dev/null +++ b/docs/sdk-python.md @@ -0,0 +1,513 @@ +--- +layout: default +title: Python SDK +nav_order: 4 +--- + +# Python SDK (`xray-py`) + +The Python SDK is how you author conversations, drive them against your +LiveKit agent, and wire the agent so its OpenTelemetry spans land in xray. +This is the authoritative reference; it is generated from — and kept in sync +with — the source under [`sdk/python/`](https://github.com/xray-eval/xray/tree/main/sdk/python). + +- Package: **`xray-py`**, import name **`xray`**, Elastic-2.0, ships `py.typed`. +- Requires **Python ≥ 3.10**. +- Everything in `__all__` is importable directly as `xray.`. + +--- + +## Install + +```bash +pip install xray-py # base — authoring + a custom Runtime +pip install "xray-py[livekit]" # the scripted + live LiveKit runtimes +pip install "xray-py[live]" # OS-mic capture for run_live (sounddevice) +``` + +Base dependencies: `httpx`, the OpenTelemetry API/SDK + OTLP/HTTP exporter, +`pydantic`, `typing-extensions`. The `[livekit]` extra pulls in `livekit` + +`livekit-api`; `[live]` adds `sounddevice` for microphone capture. + +--- + +## Public API surface + +| Export | Kind | Purpose | +|---|---|---| +| `Conversation` | dataclass | The test spec: turns + conversation-level judges. | +| `Turn` | dataclass | One step; build via `Turn.user(...)` / `Turn.agent(...)`. | +| `Assertion` | dataclass | Declarative per-turn check; 9 builder classmethods. | +| `Judge` | dataclass | Conversation-level LLM evaluator; `Judge.text_match(...)`. | +| `RecordedAudio` / `TtsAudio` | dataclass | The two user-turn audio references. | +| `RunConfig` | dataclass | Per-replay config (model, temperature, extras). | +| `run` | async fn | Orchestrate a scripted Conversation → `ReplayResult`. | +| `run_live` | async fn | Orchestrate an unscripted OS-mic session → `ReplayResult`. | +| `attach` | async context manager | Wire xray onto a LiveKit Agents entrypoint. | +| `XraySession` | class | Agent-side handle yielded by `attach`; `.turn(idx)`. | +| `SimulatedSipCall` | dataclass | `sip.*` JWT attributes for a simulated SIP participant. | +| `ReplayResult` | dataclass | The server verdict returned by `run` / `run_live`. | +| `AssertionOutcome` / `JudgeOutcome` / `TurnMetrics` | dataclass | Per-item server results. | +| `AgentResponse` / `ToolCall` / `ModelUsage` | dataclass | Runtime-captured artifacts (informational). | +| `Role` / `EvaluationStatus` | type alias | `"user" \| "agent"` and `"passed" \| "failed" \| "errored"`. | +| `format_failures` | fn | Render non-passed outcomes as a string. | +| `XrayError` / `ReplayEvaluationError` | exception | SDK / server-chain errors. | + +The runtimes and the low-level OTEL helpers live one import below the top +level: `xray.runtime.livekit.LiveKitRuntime`, `xray.runtime.livekit_live.LiveKitLiveRuntime`, +the `Runtime` ABC + protocols in `xray.runtime.base`, and `install` / +`XraySpanExporter` / `XrayBaggageSpanProcessor` in `xray.otel`. + +> There is **no `LiveKitDriver`** — the v1 LiveKit class is `LiveKitRuntime`. +> There is **no `xray.instrument` decorator** — the wiring entry point is the +> async context manager `attach`. + +--- + +## Authoring a Conversation + +```python +from xray import Assertion, Conversation, Judge, Turn + +conv = Conversation( + name="booking-happy-path", + turns=[ + Turn.user("Hi, I'd like to book a table for two at 7pm.", key="u0"), + Turn.agent( + key="a0", + assertions=( + Assertion.contains("confirmed"), + Assertion.tool_called("reserve_table"), + Assertion.max_latency_ms(2_000), + ), + ), + ], + judges=(Judge.text_match("agent confirms a reservation for two", pass_score=80),), +) +``` + +### `Conversation` + +```python +Conversation(name: str, turns: list[Turn], judges: tuple[Judge, ...] = (), live: bool = False) +``` + +A `Conversation`'s identity is a SHA-256 content hash over the canonical spec +(turns + per-turn assertions + judges, with any `RecordedAudio` bytes folded +in by their sha256). `name` is a free-form display label — renaming the same +spec re-attaches Replays to the same Conversation row; editing any turn, +assertion, judge, or WAV forks a new Conversation. The server computes the +hash; the SDK never hashes anything. Construction raises `ValueError` on an +empty `name`, or on empty `turns` unless `live=True`. + +### `Turn` + +```python +Turn.user(text: str, *, key=None, audio: AudioRef | None = None, assertions=()) -> Turn +Turn.agent(*, key=None, assertions=()) -> Turn +``` + +`Turn.agent` takes **no `text`** — the agent's text is observed at runtime and +transcribed server-side, not declared. A user turn with no `audio` is sent as +a server-side TTS marker (see [Audio references](#audio-references)). + +### Assertions + +All nine builders are `Assertion` classmethods and validate their arguments at +construction (fail-fast `ValueError`). All run **server-side** during the +`evaluate-replay` stage. + +```python +Assertion.contains(text, *, case_insensitive=True) +Assertion.not_contains(text, *, case_insensitive=True) +Assertion.equals(text, *, case_insensitive=True, trim=True) +Assertion.regex(pattern, *, flags="") +Assertion.tool_called(name) +Assertion.tool_not_called(name) +Assertion.tool_args_match(name, args) # args: dict[str, JsonValue] +Assertion.max_latency_ms(max_ms) # max_ms >= 1 +Assertion.max_ttft_ms(max_ms) # max_ms >= 1 +``` + +| Kind | Checks | +|---|---| +| `contains` / `not_contains` | The agent transcript does / does not contain `text`. | +| `equals` | The transcript equals `text` (optionally trimmed / case-insensitive). | +| `regex` | The transcript matches `pattern` (with optional `flags`). | +| `tool_called` / `tool_not_called` | A tool named `name` was / was not called in the turn. | +| `tool_args_match` | A `name` call's arguments match the given subset. | +| `max_latency_ms` | The agent responded within `max_ms` of the user turn ending. | +| `max_ttft_ms` | The model's time-to-first-chunk was within `max_ms`. | + +The tool/TTFT assertions depend on span→turn attribution from the audio +timeline. If the runtime uploaded no recording anchor, they come back +**`errored`** (not failed) — xray can't place spans on the timeline without +it. `LiveKitRuntime` always reports the anchor; a custom `Runtime` must too +(see [Runtimes](#runtimes)). + +### Judges + +```python +Judge.text_match(reference: str, *, rubric: str | None = None, pass_score: int = 70) -> Judge +``` + +A conversation-level LLM judge: the server asks the configured judge model to +score the full transcript against `reference` (optionally guided by `rubric`) +on a 0–100 scale, passing iff `score >= pass_score`. `text_match` is the only +judge kind in v1. Validation raises `ValueError` on an empty `reference`, an +empty `rubric`, or a `pass_score` outside `0..100`. + +### Audio references + +```python +RecordedAudio(path: str) # an on-disk WAV: 48 kHz, mono, 16-bit +TtsAudio(voice_id: str | None = None) +``` + +A user turn's `audio` is one of these (or omitted, which is equivalent to +`TtsAudio()`): + +- **`RecordedAudio`** — a real WAV you already have. Its bytes are uploaded as + a multipart file part with the conversation and folded into the hash. +- **`TtsAudio`** — a marker that the **server** should synthesize the turn. + The xray server synthesizes at conversation-upsert time using the provider + it's configured with (`XRAY_TTS_PROVIDER`, voice from `XRAY_TTS_VOICE` or the + per-turn `voice_id`), content-addresses the WAV, and folds its sha256 into + the conversation hash. **The SDK does no TTS** — no provider key in the SDK + process. `run(...)` simply fetches the synthesized bytes back over HTTP + before driving the room. + +Convert a WAV to the required format with +`ffmpeg -i in.wav -ar 48000 -ac 1 -sample_fmt s16 out.wav`. For voices xray +doesn't host (Cartesia, ElevenLabs, Deepgram, …), synthesize externally and +pass the result as `RecordedAudio`. + +--- + +## `RunConfig` + +```python +RunConfig(model: str | None = None, temperature: float | None = None, + extra: dict[str, JsonValue] = {}) +``` + +Per-replay configuration carried to the server on `POST /v1/replays`. `extra` +keys are flattened to the top level on the wire so the compare UI diffs them +as first-class columns; `model` / `temperature` are omitted when `None`. + +--- + +## Running + +### `run` — scripted + +```python +async def run(*, conversation: Conversation, runtime: Runtime, + xray_url: str = "http://localhost:8080", + run_config: RunConfig | None = None) -> ReplayResult +``` + +Fully keyword-only and async — wrap in `asyncio.run(...)` for a sync harness. +There is no sync `run`. End to end it: + +1. Checks every `RecordedAudio` file exists locally. +2. POSTs the Conversation to `/v1/conversations` (multipart: a `spec` JSON + part + one file part per `RecordedAudio` turn) and reads back the hash. +3. Prefetches every user turn's audio (the bytes the server synthesized or + stored) — before creating the replay, so a failure leaves no orphan row. +4. POSTs the Replay to `/v1/replays` (`{conversation_hash, run_config?}`) and + reads back its `id`. +5. Drives the runtime: binds the replay context, injects user audio, installs + the OTEL pipeline pointed at `xray_url`, attaches replay baggage, runs the + runtime, then force-flushes spans. +6. Uploads the stereo mixdown WAV to `/v1/replays/:id/audio` with the + `X-Recording-Started-At` header, then POSTs `/v1/replays/:id/analyze`. +7. Streams `/v1/replays/:id/events` until `evaluation_complete` (→ returns + `ReplayResult`) or `failed` (→ raises `ReplayEvaluationError`). + +```python +import asyncio +import xray +from xray import Assertion, Conversation, Judge, RunConfig, Turn +from xray.runtime.livekit import LiveKitRuntime + +async def main() -> None: + conv = Conversation(name="booking", turns=[...], judges=(...,)) + runtime = LiveKitRuntime(url=..., api_key=..., api_secret=..., room="booking-test") + result = await xray.run( + conversation=conv, + runtime=runtime, + xray_url="http://localhost:8080", + run_config=RunConfig(model="gpt-4o", temperature=0.5), + ) + assert result.passed, xray.format_failures(result) + +asyncio.run(main()) +``` + +Per-assertion and per-judge failures **do not raise** — they're outcomes on +the result. `assert result.passed, xray.format_failures(result)` is the pytest +idiom. Only driver-side faults (`XrayError` subclasses) and server-chain +crashes (`ReplayEvaluationError`) raise. + +### `run_live` — unscripted (OS mic) + +```python +async def run_live(*, runtime: Runtime, xray_url: str = "http://localhost:8080", + name: str | None = None, + run_config: RunConfig | None = None) -> ReplayResult +``` + +For talking to the agent yourself instead of replaying a scripted WAV. There's +no authored `Conversation`: `run_live` builds an empty `live=True` Conversation +(named `name` or `live-`; the server salts the hash so each session +is its own row), records your mic + the agent's audio, and analyzes the result +the same way. SIGINT (Ctrl-C) ends the session and uploads it. The returned +`ReplayResult` has empty `assertions` / `judges` and `passed=True`, but +populated `metrics`. + +```python +from xray.runtime.livekit_live import LiveKitLiveRuntime + +runtime = LiveKitLiveRuntime(url=..., api_key=..., api_secret=..., room=...) +result = await xray.run_live(runtime=runtime, xray_url="http://localhost:8080") +``` + +--- + +## Wiring the agent + +### `attach` + +```python +@asynccontextmanager +async def attach(ctx, *, service_name: str | None = None, + endpoint: str | None = None, + bind_timeout_s: float = 10.0) -> AsyncGenerator[XraySession | None, None] +``` + +Wrap your LiveKit Agents worker entrypoint with `attach`. It is an **async +context manager, not a decorator** — a decorator wrapper breaks LiveKit +Agents' forkserver pickling. + +```python +import xray +from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli + +async def entrypoint(ctx: JobContext) -> None: + await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY) + async with xray.attach(ctx, service_name="my-agent") as session: + # session is None when no xray-tagged participant joined (i.e. prod). + await your_agent.run(ctx, session=session) + +cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint)) +``` + +- Call it **after** `ctx.connect(...)` — before connect, the room has no + remote participants to scan. +- **Endpoint resolution:** the explicit `endpoint=` argument, else the + `XRAY_OTLP_ENDPOINT` environment variable, else none. With no endpoint, + `attach` still binds baggage in-process but installs no OTLP exporter. (In + production no participant carries the `xray` attribute, so `attach` yields + `None` and is effectively a no-op.) +- On block exit it detaches baggage and force-flushes the tracer provider so + spans land before the worker shuts down. + +`ctx` is duck-typed — any object exposing `.room.remote_participants` plus +`.on` / `.off` event hooks works; the SDK does not import `livekit.agents`. + +### Replay-context propagation + +The replay context travels on the joining participant's **JWT `xray` +attribute** (read via `participant.attributes["xray"]`). No room or participant +metadata is set, and no `can_update_own_metadata` grant is required. The JSON +payload is exactly: + +```json +{ "replay_id": "...", "conversation_hash": "...", "modality": "voice" } +``` + +`attach` reads it, sets OTEL baggage (`xray.replay.id`, +`xray.conversation.hash`, `xray.modality`), and the bundled span processor +lifts that baggage onto every span at start. The server routes spans by +`xray.replay.id`. + +### `XraySession` + +`attach` yields an `XraySession` (or `None`). It exposes read-only +`replay_id` / `conversation_hash` / `modality`, plus one helper: + +```python +async with session.turn(idx, key=None): + ... # emits an xray.turn span + scopes xray.turn.* baggage for this turn +``` + +### Low-level OTEL helpers (`xray.otel`) + +If you wire OpenTelemetry manually instead of using `attach`: + +- `install(*, endpoint, tracer_provider=None) -> TracerProvider` — idempotent; + registers the baggage span processor + a batch exporter. +- `XraySpanExporter` — POSTs **OTLP/JSON** to `${endpoint}/v1/otlp/v1/traces`. +- `XrayBaggageSpanProcessor` — lifts the `xray.*` baggage onto every span. + +--- + +## Runtimes + +A `Runtime` drives one Conversation against your agent and produces the audio +recording. `LiveKitRuntime` is the v1 implementation; you can subclass +`Runtime` for any other transport. + +### The `Runtime` ABC + +```python +class Runtime(ABC): + async def run(self, conversation: Conversation) -> RuntimeResult: ... + async def aclose(self) -> None: ... +``` + +```python +@dataclass +class RuntimeResult: + responses: list[AgentResponse] = field(default_factory=list) + full_audio_path: str | None = None + full_transcript: str | None = None + recording_started_at_epoch: float | None = None # Unix seconds of audio sample 0 +``` + +`recording_started_at_epoch` is what `run(...)` turns into the +`X-Recording-Started-At` upload header. **A runtime that produces audio MUST +report it** — omit it and span→turn attribution is skipped, so every +`tool_called` / `tool_not_called` / `tool_args_match` / `max_ttft_ms` +assertion comes back `errored`. + +Optional structural protocols the orchestrator probes for (`@runtime_checkable`): + +- `RuntimeBindable` — `bind(*, replay_id, conversation_hash)`. +- `UserAudioInjectable` — `inject_user_audio(audio: Mapping[int, bytes])`. +- `StoppableRuntime` — `request_stop()` (used by `run_live` on SIGINT). + +### `LiveKitRuntime` + +```python +LiveKitRuntime( + url: str, api_key: str, api_secret: str, room: str, + identity: str = "xray-driver", + agent_join_timeout_s: float = 30.0, + agent_turn_timeout_s: float = 30.0, + cache_root: Path = ~/.cache/xray-py, + mixdown_dir: Path | None = None, + simulated_sip: SimulatedSipCall | None = None, +) +``` + +Joins the room as a user-side participant, plays the per-turn user PCM, captures +the agent's audio + transcripts, and writes a wall-clock-aligned stereo WAV +(left = user, right = agent) at 48 kHz / 16-bit. Implements `bind`, +`inject_user_audio`, `run`, and `aclose`; `run` raises `RuntimeBindError` if +called before `bind`. Used with `xray.run`. + +### `LiveKitLiveRuntime` + +```python +LiveKitLiveRuntime( + url: str, api_key: str, api_secret: str, room: str, + identity: str = "xray-driver", + agent_join_timeout_s: float = 30.0, + agent_audio_timeout_s: float | None = None, + play_agent_audio: bool = True, + cache_root: Path = ~/.cache/xray-py, + mixdown_dir: Path | None = None, + simulated_sip: SimulatedSipCall | None = None, +) +``` + +Powers `run_live`: streams the OS microphone (needs the `[live]` extra), +publishes mic frames, captures and optionally plays the agent's audio, and +writes a live stereo mixdown. Set `play_agent_audio=False` (or +`XRAY_LIVE_NO_PLAYBACK=1` in the example) for a record-only run. It emits no +`xray.turn` spans — turn boundaries come from server-side VAD. + +### `SimulatedSipCall` + +```python +SimulatedSipCall( + caller_phone=None, trunk_phone=None, call_id=None, call_id_full=None, + call_status=None, rule_id=None, trunk_id=None, + extra_attrs: Mapping[str, str] = {}, +) +``` + +Pass to a runtime's `simulated_sip=` to make the driver join as a simulated SIP +participant: the driver mints the JWT with `with_kind("sip")` plus the `sip.*` +attributes (`sip.phoneNumber`, `sip.trunkPhoneNumber`, `sip.callID`, +`sip.callStatus`, …). `call_status` is one of `"active"`, `"automation"`, +`"dialing"`, `"hangup"`, `"ringing"`. An all-empty object raises `ValueError` +(use `simulated_sip=None` for a non-SIP run), as does an `"xray"` key in +`extra_attrs`. + +--- + +## `ReplayResult` and outcomes + +```python +@dataclass(frozen=True) +class ReplayResult: + replay_id: str + conversation_hash: str + passed: bool + assertions: tuple[AssertionOutcome, ...] + judges: tuple[JudgeOutcome, ...] + metrics: tuple[TurnMetrics, ...] +``` + +`passed` is the aggregate: `True` iff every assertion **and** every judge ran +to `"passed"` — `"errored"` counts as not-passed. + +```python +AssertionOutcome(turn_idx: int, assertion_idx: int, kind: str, + status: EvaluationStatus, message: str | None) +JudgeOutcome(judge_idx: int, kind: str, status: EvaluationStatus, + score: int | None, reason: str | None) +TurnMetrics(turn_idx: int, role: Role, agent_response_ms: int | None, + interrupted: bool) +``` + +`EvaluationStatus` is `"passed" | "failed" | "errored"`. `format_failures(result)` +renders just the non-passed assertion + judge outcomes (or +`"all assertions and judges passed"`). + +`AgentResponse` / `ToolCall` / `ModelUsage` are informational records the +runtime captured during the run; they are **not** what evaluation reads +(evaluation runs server-side from the declared catalog + the OTLP spans). + +--- + +## Errors + +`XrayError` is the base SDK exception; every subclass carries a +`failure_reason`. The ones a test might catch: + +| Class | `failure_reason` | When | +|---|---|---| +| `RuntimeBindError` | `driver_aborted` | `run()` called before `bind()`. | +| `AgentNotJoinedError` | `agent_not_joined` | The agent participant didn't join in time. | +| `AudioMissingError` | `audio_missing` | A user turn's audio can't be materialized, or a live session captured no frames. | +| `AudioTooLargeError` | `driver_aborted` | The mixdown exceeds 50 MiB. | +| `MixdownError` | `driver_aborted` | Writing the WAV mixdown failed. | +| `LiveKitDependencyError` / `LiveDependencyError` | `driver_aborted` | The `[livekit]` / `[live]` extra isn't installed. | +| `MicCaptureError` / `SpeakerPlaybackError` | `driver_aborted` | The OS mic / speaker can't be opened (live sessions). | +| `XrayServerError` | `driver_aborted` | An HTTP error from the server before the replay row exists. | +| `ReplayEvaluationError` | the failing stage | The server's analyze chain crashed before producing a verdict (`transcription_failed` / `metrics_failed` / `evaluation_failed`, …). Carries `replay_id`. | + +Driver-side failures map to a `PATCH /v1/replays/:id` so the replay records why +it stopped. Lifecycle transitions during the analyze chain are server-owned. + +--- + +## See also + +- [`integrate.md`](./integrate.md) — the end-to-end walkthrough. +- [`wire-contract.md`](./wire-contract.md) — what the agent's spans must look like. +- [`architecture.md`](./architecture.md) — how the server turns a run into a verdict. diff --git a/docs/specs/0001-timeline-clock-alignment.md b/docs/specs/0001-timeline-clock-alignment.md deleted file mode 100644 index 794c13b..0000000 --- a/docs/specs/0001-timeline-clock-alignment.md +++ /dev/null @@ -1,275 +0,0 @@ -# 0001 · Timeline clock alignment + timeline-native span model - -Status: **implemented** (server + SDK + client; verified: `bun test` 680 pass, `tsc` clean, biome clean, SDK `pytest` 93 pass, `pyright --strict` clean) -Branch: `feat/metrics-eval` - -> Spec numbering: no prior spec exists on this branch; `0001` chosen to -> match the per-feature convention used on earlier branches. Renumber if -> a global sequence is adopted. - ---- - -## 1 · The bug, precisely - -xray computes per-turn metrics and attributes OTLP spans (`tool_calls`, -`model_usage`) to turns by mapping each span's wall-clock timestamp onto -the **audio timeline** (ms from the recording's `t=0`). The map it uses -is wrong: - -```ts -// analyze-replay.processor.ts:346 & calculate-metrics.processor.ts:259 -offsetMs = Date.parse(span.started_at) − Date.parse(replays.started_at) -``` - -`replays.started_at` is stamped at **replay-row creation** (POST -`/v1/replays`, `replays.service.ts:65`), which happens **before** the -driver connects the LiveKit room, waits for the agent to join -(`livekit.py:219`), publishes the track, and begins the first turn. -Audio `t=0` is `min(segment.started_at)` — the first turn's driver-clock -`time.time()` (`livekit.py:581`). The two instants differ by the entire -connect + agent-join + publish latency. - -### Evidence (authentic snapshot) - -`snapshot/xray.db`, replay `7b8e2770…`: - -| quantity | wall-clock | -|---|---| -| `replays.started_at` | `14:31:28.688` | -| first `xray.turn` span ≈ audio `t=0` | `14:31:31.023` | - -**Gap = 2335 ms**, run-dependent and unbounded (scales with agent -cold-start). Worked example — tool call `get_current_year` at -`14:31:37.265`, VAD turn windows `t0[360,2190) t1[2550,4680) -t2[7110,11070)`: - -- code offset `37265 − 28688 = 8577` → lands in `t2` voice window -- true offset `37265 − 31023 = 6242` → lands in `t2`'s pre-voice gap - -The two errors (wrong origin; voice-window-only matching) happen to -partially cancel here. They do not in general. - -### Why the server can't fix it alone - -At the driver→server boundary the driver observes a **join**: -`(wall_clock ↔ audio_sample ↔ turn_structure)`. It ships raw audio bytes -(no timestamps) + raw OTLP spans (wall-clock, mixed clock domains) and -**discards the join**. `started_at` is a server-clock stamp of an -unrelated event; it does not contain the join. The fix must **preserve -the join across the boundary** — the driver is the only party that knows -audio `t=0` in wall-clock terms. - -### Clock domains in play - -| Domain | Owns | Notes | -|---|---|---| -| Server | `replays.started_at` | unrelated to audio; row-creation time | -| Driver (test runner) | the mixdown, audio `t=0`; `xray.turn` spans | the join lives here | -| Agent under test | `gen_ai.*` / tool spans | separate process; the timestamps we attribute | - -Runs are short → clock-rate drift is negligible → the correction is a -pure **offset**, not an affine slope. - ---- - -## 2 · Decisions (resolved with the maintainer) - -| # | Decision | Choice | Rationale | -|---|---|---|---| -| D1 | Agent-clock correlation | **Co-located anchor.** Driver ships audio `t=0`; assume agent clock ≈ driver clock. **No** out-of-range flagging. | Common setup is one host. Spans legitimately fire outside any turn's audio (prep, speculative gen), so "outside the audio" is not an error condition. | -| D2 | Turn source | **VAD-authoritative, unchanged.** | Required for live mode; out of scope to revisit here. | -| D3 | Anchor granularity | **Single recording-start anchor.** | One offset map for the whole run; driver already knows `t=0`. | -| D4 | `ttft_ms` meaning | **Model TTFT from GenAI semconv, optional.** | The value is a same-clock delta — correlation-free. Treated as one more optional extracted field, no special pipeline. | -| D5 | Anchor wire surface | **`X-Recording-Started-At` header on POST `/audio`.** | The anchor is a property *of the audio*; persisted once at upload, read by the chain. | -| D6 | Refactor scope | **Drop stored `turn_idx`; delete `backfillTurnIdx`.** Spans carry a read-derived `audio_offset_ms`; display by timeline overlap; assertions compute per-turn membership at eval time. | A stored `turn_idx` denormalizes a spatial fact the timeline already encodes, and forces one lossy global attribution rule. | -| D7 | Per-turn assertion window | **`[turnStartMs, turnEnd)`** — the agent turn's *own* window (tiles the timeline). | Strict: a too-early (speculative during the user's turn) or too-late tool call lands in a neighbour tile → flagged. Speculative-execution leniency deferred to an explicit eval-SDK opt-in. | - ---- - -## 3 · Design - -### 3.1 The anchor - -- New column **`replays.recording_started_at`** (`text`, nullable, - ISO-8601 UTC). -- POST `/audio` reads header **`X-Recording-Started-At`**, Valibot-validated - (ISO datetime), persists it on the replay row alongside `audio_path`. -- **Absent header** (legacy SDK / unset): store `null`. Degrade gracefully - — no offset, no per-turn tool attribution (see §3.4). Never fall back - to `started_at`. New SDK always sends it. - -### 3.2 Unified timeline — `audio_offset_ms` - -A single pure helper maps wall-clock → audio offset: - -```ts -// audio_offset_ms = parse(startedAtIso) − parse(recordingStartedAtIso) -// → number | null (null if either timestamp is missing/unparseable) -``` - -Derived **at read** for every `span` / `tool_call` / `model_usage` from -data already stored (`started_at` on the row + `recording_started_at` on -the replay). **Not** a stored column — single source of truth, cannot -drift. Used by both the response builder (inspector timeline) and the -assertion evaluator. - -The inspector renders all spans + VAD turns on one ms axis; turn overlap -is **visual**, not a stored foreign key. A span in a gap (prep / -speculative) renders in the gap and reads correctly. - -### 3.3 TTFT — optional, no special handling - -- `gen-ai-semconv.ts` extracts `gen_ai.response.time_to_first_chunk` - (seconds, float; semconv stability *Development* — pin against the - emitting instrumentation's semconv version) → `ttftMs = round(s*1000)`. - Added to `ExtractedModelUsage`. Null when the span doesn't carry it - (the common case today — the example v2v agent emits none). -- New column **`model_usage.ttft_ms`** (`integer`, nullable). Written by - `persistExtracted`. Rides the timeline as a model-call attribute. -- **Removed** from `replay_metrics` entirely. - -### 3.4 Assertions — eval-time membership - -**Scope: this window governs `tool_called` / `tool_not_called` / -`tool_args_match` / `max_ttft_ms` evaluation only.** Every recognized -`tool_call` / `model_usage` / span is still recorded *unconditionally* -(`otlp.service.ts` `persistExtracted` — no turn association at rest, since -D6 drops `turn_idx`) and rendered on the timeline at its true -`audio_offset_ms`, in-turn or not. The window never filters storage or -display: a speculative tool call that fires during the user's turn is -still stored and still shown — it just doesn't count toward the agent -turn's assertion. - -`buildAssertionContext` (`evaluate-replay.processor.ts`) computes each -turn's tool/model rows on demand: - -- A `tool_call` / `model_usage` row belongs to turn `N` iff its - `audio_offset_ms ∈ [turnStartMs_N, turnEndMs_N)` — the turn's **own** - window from `deriveTurns`, where `turnStartMs` = the previous - (opposite-role) turn's `voiceEndMs` (0 if none) and `turnEndMs` = this - turn's `voiceEndMs`. Since `turnStartMs_N = voiceEndMs_{N-1}`, the - windows **tile** the timeline with no gaps or overlap — every call maps - to **exactly one** turn. -- **Strict by design.** A call that fires *before* the user stopped - (speculative-during-user-turn) or *after* the agent finished lands in a - neighbouring turn's tile → **not** counted for turn `N` → `tool_called` - flags it. A mistimed tool call is a real agent bug worth surfacing. The - pre-voice "thinking" gap stays in-turn (it sits in `N`'s tile, after the - user stopped). -- `max_ttft_ms` sources `ttftMs` from the **earliest in-window** - `model_usage` row (the first LLM call's perceived first-chunk latency). -- **No `recording_started_at` → no offsets → tool/ttft assertions - `errored`** with a clear message (`"no recording anchor; cannot - attribute spans to turns"`). Honest, not silently failing. - -> Future opt-in: if speculative execution outside the bot turn proves -> common, add an explicit eval-SDK knob (e.g. an assertion param -> `allow_speculative`, or a widened window) rather than loosening the -> default. Strict-by-default; widen on demand. - -### 3.5 calculate-metrics simplification - -With TTFT gone from `replay_metrics`, the stage no longer reads `spans` -or `recording_started_at`. It computes only audio-derived, single-frame -metrics — both already correct: - -- `agent_response_ms` = `voiceStartMs − priorUserVoiceEndMs` -- `interrupted` / `interruption_start_ms` - -`computeMetrics` loses its `ttftSpans` + `replayStartMs` params and the -`ttftFor` helper is deleted. - ---- - -## 4 · Change surface - -### Deletions - -- `analyze-replay.processor.ts`: `backfillTurnIdx`, `turnIdxForStartedAt`, - and all `tool_calls` / `model_usage` writes from this stage. -- `calculate-metrics.processor.ts`: `ttftFor`; `ttftSpans` + `replayStartMs` - plumbing. -- `model_usage.turn_idx`, `tool_calls.turn_idx` columns. -- `replay_metrics.ttft_ms` column. - -### Server - -- `store/schema.ts` + migration: `+replays.recording_started_at`, - `+model_usage.ttft_ms`, `−*.turn_idx`, `−replay_metrics.ttft_ms`. -- `audio.router.ts` / `audio.service.ts` / `audio.types.ts`: parse + persist - the header. -- `replays/timeline.ts` (new small slice): the `audio_offset_ms` helper + - the membership-window predicate. Co-located test. -- `otlp/vocabularies/gen-ai-semconv.ts`: extract `time_to_first_chunk`. -- `otlp.service.ts` (`persistExtracted`): write `model_usage.ttft_ms`; stop - writing `turn_idx`. -- `evaluate-replay.processor.ts`: eval-time membership; ttft from model_usage. -- `replays.service.ts` + `replays.types.ts`: response shapes (§5). - -### Wire contract (breaking) - -| Schema | Change | -|---|---| -| `ToolCallResponse` | −`turn_idx`, +`audio_offset_ms` (nullable) | -| `ModelUsageResponse` | −`turn_idx`, +`audio_offset_ms` (nullable), +`ttft_ms` (nullable) | -| `TurnMetricsResponse` | −`ttft_ms` | -| POST `/audio` | +`X-Recording-Started-At` request header (optional) | - -### SDK - -- `RuntimeResult`: +`recording_started_at_epoch: float | None` (the mixdown - `t=0`, already computed in `write_stereo_mixdown` / `write_live_mixdown`). -- `livekit.py`: return it from `run()`. -- `orchestrator.py` `_upload_replay_audio`: send the `X-Recording-Started-At` - header (ISO from the epoch). - -### Client - -- Inspector: render spans on the shared timeline by `audio_offset_ms`; - drop the per-turn `turn_idx` grouping for tool/model rows; surface - `model_usage.ttft_ms` when present. - ---- - -## 5 · Test plan (TDD — tests land red first) - -- **Regression (the decisive one):** evaluate-replay with - `recording_started_at` deliberately ≠ `started_at` (a multi-second gap) + - spans whose wall-clock matches the agent's real emit time → assert tool - calls attribute to the correct turn. This is the case **no current test - exercises** — today's fixtures bake `span = started_at + offset` - (`calculate-metrics.processor.test.ts:89`), encoding the bug. -- gen-ai-semconv: `time_to_first_chunk` → `ttft_ms` (present / absent / unit). -- timeline helper: offset math; null when either timestamp missing. -- membership tiling `[turnStartMs, turnEndMs)` — exactly-one-turn; a - speculative-during-user-turn call lands in the user tile and is - **flagged** for the agent turn (`tool_called` fails); no-anchor → - `errored`. -- audio router: header parse (valid / malformed / absent). -- store migration test. -- Re-point / simplify calculate-metrics + analyze-replay tests (backfill gone). - ---- - -## 6 · Risks & accepted assumptions - -- **Co-located clocks (D1).** Agent on a different, non-NTP-synced host → - timeline positions and tool attribution skew, **silently** (no flag, per - D1). Documented; revisit with audio cross-correlation or trace-context - propagation if distributed runs become real. -- **VAD real-audio accuracy.** Every remaining audio-frame metric - (`agent_response_ms`, `interrupted`) and every assertion window rests on - VAD, currently calibrated on synthetic sines only. Unchanged here, but now - the *sole* dependency. Separate follow-up. -- **TTFT availability.** Depends on the dev's instrumentation emitting the - experimental `gen_ai.response.time_to_first_chunk`. Null otherwise. - Accepted (D4). -- **No anchor (legacy SDK).** Timeline + tool/ttft assertions degrade to - null / `errored`. New SDK always sends the header. - -## 7 · Out of scope - -- Turn-source redesign (driver-declared / hybrid) — D2. -- Distributed-clock correlation (cross-correlation / trace propagation). -- `xray.stage.stt` / `xray.stage.tts` → stage-latency metrics. -- Real-audio VAD calibration. diff --git a/docs/wire-contract.md b/docs/wire-contract.md new file mode 100644 index 0000000..55c21d6 --- /dev/null +++ b/docs/wire-contract.md @@ -0,0 +1,209 @@ +--- +layout: default +title: Wire contract (OTLP) +nav_order: 5 +--- + +# OTLP wire contract + +xray ingests your agent's OpenTelemetry traces through one endpoint and turns +recognized spans into structured `tool_calls` / `model_usage` rows. This is the +contract: what the endpoint accepts, how a span is routed to a Replay, and +which span shapes are recognized. It's derived from +[`src/server/otlp/`](https://github.com/xray-eval/xray/tree/main/src/server/otlp). + +You don't have to emit xray-specific spans. Any agent already instrumented with +the OTel GenAI semantic conventions (`gen_ai.*`) or Langfuse lights up +automatically. The `xray.*` vocabulary is optional and additive. + +--- + +## Endpoint + +| | | +|---|---| +| **Path** | `POST /v1/otlp/v1/traces` | +| **Content types** | `application/json` and `application/x-protobuf` (both standard OTLP `ExportTraceServiceRequest`). | +| **Success** | `200` with `{ "partialSuccess": { "rejectedSpans": N } }`. | + +`xray.attach`'s exporter posts OTLP/JSON; the stock OTel HTTP exporter's +protobuf default works too. A `Content-Type` with parameters +(`application/json; charset=utf-8`) is matched correctly. + +### Limits + +| Cap | Value | Behaviour on exceed | +|---|---|---| +| Body size | 4 MiB | `413 body_too_large`. | +| Spans per request | 512 | `400 too_many_spans_per_request` — the whole request is rejected. | +| Spans per replay | 5,000 | Spans over the cap are counted in `rejectedSpans`; in-cap spans in the same batch still persist. No error. | + +Other failures map to `400 invalid_otlp_body` (malformed / schema-invalid body), +`415 unsupported_content_type`, or `500 internal_error`. + +### Idempotency + +Spans are de-duplicated on `(replay_id, span_id)`. Re-sending a span already +stored is a no-op — it isn't re-counted against the cap and its extracted rows +aren't re-processed. Safe to retry a batch. + +--- + +## Routing and the trust boundary + +Every span is routed to a Replay by the **`xray.replay.id`** attribute. A +**span-level** value takes precedence; the **resource-level** value is the +fallback. (`attach` sets it as baggage, which the span processor lifts onto +every span, so in practice it's present at the span level.) + +The receiver is a **filter, not a gate**. A span is silently dropped (counted +in `rejectedSpans`, never an error) when: + +1. it carries no `xray.replay.id` (no replay context — e.g. the agent running + in production); +2. the `xray.replay.id` names a Replay that doesn't exist; or +3. its vocabulary isn't recognized. + +This is the trust boundary: **the OTLP receiver never creates Conversation or +Replay rows.** It only reads existing ones. Replay rows are created exclusively +by the SDK control plane, *before* the agent emits its first span — which is +what makes "unknown replay id → drop" safe rather than lossy. + +### Timestamps + +`startTimeUnixNano` / `endTimeUnixNano` are converted to ISO-8601 and stored as +each row's `started_at` / `ended_at`. These feed the audio-timeline turn +attribution described below. + +--- + +## The three vocabularies + +Each span is run through an ordered registry; the **first** vocabulary that +recognizes it wins. Order is fixed: + +1. `xray` +2. `gen_ai` (OTel GenAI semconv) +3. `langfuse` + +A vocabulary match can emit a `tool_calls` row, a `model_usage` row, or +neither — but **every** recognized span is also stored raw in the `spans` +table (tagged with the matching vocabulary) for the inspector's timeline. + +### 1 · `xray` + +Recognizes exactly three span names — an exact-match set, **not** a prefix +wildcard: + +- `xray.turn` +- `xray.stage.stt` +- `xray.stage.tts` + +These land in the raw `spans` table only; they produce no `tool_calls` / +`model_usage` rows. Turn boundaries come from server-side VAD, and assertion / +judge outcomes come from the declared catalog — not from these spans. Any other +`xray.*` name (e.g. `xray.stage.llm`) is unrecognized and dropped. + +> `xray.assertion` and `xray.judge` are **not** recognized. Evaluation runs +> server-side from the `Assertion` / `Judge` catalog declared on the +> Conversation, so driver-emitted assertion/judge spans are intentionally +> ignored. + +### 2 · `gen_ai` (OTel GenAI semantic conventions) + +Dispatches on **`gen_ai.operation.name`** (a span also counts as GenAI if any +attribute key starts with `gen_ai.`, or its name starts with `chat`, +`text_completion`, or `execute_tool`). + +**`execute_tool` → `tool_calls` row:** + +| Field | From | +|---|---| +| `name` | `gen_ai.tool.name` (fallback: span name minus the `execute_tool ` prefix) | +| `args_json` | `gen_ai.tool.arguments` | +| `result_json` | `gen_ai.tool.result` | +| `latency_ms` | span `end − start` | + +**`chat` or `text_completion` → `model_usage` row:** + +| Field | From | +|---|---| +| `provider` | `gen_ai.system` | +| `model` | `gen_ai.response.model` (fallback `gen_ai.request.model`) | +| `input_tokens` / `output_tokens` | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | +| `total_tokens` | sum of the two (null only if both absent) | +| `ttft_ms` | `gen_ai.response.time_to_first_chunk` — interpreted as **seconds**, converted to ms | +| `latency_ms` | span `end − start` | + +Any other operation (e.g. `embeddings`) is stored as a raw `gen_ai` span with +no extracted row. + +> Earlier docs referred to `gen_ai.tool` and `gen_ai.client.operation`. Those +> are not what the code matches on — the dispatch key is +> `gen_ai.operation.name` with values `execute_tool` / `chat` / +> `text_completion`. + +### 3 · `langfuse` + +Recognizes any span carrying a `langfuse.`-prefixed attribute. The observation +type is read from `langfuse.observation.type` (fallback `langfuse.type`). + +**`generation` → `model_usage` row:** + +| Field | From | +|---|---| +| `provider` | `langfuse.observation.provider` | +| `model` | `langfuse.observation.model.name` | +| `input_tokens` / `output_tokens` | `langfuse.observation.usage_details.input` / `.output` | +| `total_tokens` | `langfuse.observation.usage_details.total` (read directly) | +| `ttft_ms` | always `null` (not sourced from Langfuse) | + +**`tool` → `tool_calls` row:** + +| Field | From | +|---|---| +| `name` | `langfuse.observation.name` (fallback: span name) | +| `args_json` / `result_json` | `langfuse.observation.input.value` / `.output.value` | + +Other observation types (`event`, `span`, `score`, unset) are stored as raw +`langfuse` spans with no extracted row. + +--- + +## What lands where + +| Table | Written for | When | +|---|---|---| +| `spans` | every accepted span | always, regardless of vocabulary | +| `tool_calls` | gen_ai `execute_tool`, langfuse `tool` | a tool was observed | +| `model_usage` | gen_ai `chat` / `text_completion`, langfuse `generation` | an LLM call was observed | + +### Turn attribution is derived, not stored + +`tool_calls` and `model_usage` carry only `replay_id` and (nullable) `span_id` +— there is **no `turn_idx` column** on them. A row's turn membership is +computed at evaluation/read time by mapping its wall-clock `started_at` onto the +audio timeline: + +``` +audio_offset_ms = started_at − replays.recording_started_at +``` + +and testing it against the turn windows derived from VAD. The +`recording_started_at` origin is set by the driver's audio upload (the +`X-Recording-Started-At` header), never by this OTLP path. With no anchor, the +timeline-dependent assertions (`tool_called`, `tool_not_called`, +`tool_args_match`, `max_ttft_ms`) return `errored`. The origin must be the +audio sample-0 wall-clock (the `X-Recording-Started-At` header), never the +replay row's creation time (which precedes the recording). + +--- + +## Adding a vocabulary + +Each vocabulary is one file in +[`src/server/otlp/vocabularies/`](https://github.com/xray-eval/xray/tree/main/src/server/otlp/vocabularies) +exporting a pure `match(span, resource)` function, plus one line in +`registry.ts`. Test it against synthetic projected spans with the slice's +test-utils — no network. See [`architecture.md`](./architecture.md) and the +contributing guide. diff --git a/examples/livekit-voice-agent/README.md b/examples/livekit-voice-agent/README.md index aed6fe3..f525446 100644 --- a/examples/livekit-voice-agent/README.md +++ b/examples/livekit-voice-agent/README.md @@ -14,7 +14,9 @@ end-to-end. Demonstrates the one-line `xray.attach(ctx)` integration. ├── agent/main.py ← the one xray.attach() call ├── driver/test_e2e.py ← pytest; drives one scripted Replay ├── driver/live_session.py ← interactive; talk to the agent over the mic -└── fixtures/user_turn_1.wav +└── fixtures/ + ├── user_turn_1.wav ← scripted user utterance (48 kHz mono) + └── agent_greeting.wav ← greeting the agent plays via session.say ``` ## Quickstart diff --git a/sdk/python/.claude/rules/typed-boundaries.md b/sdk/python/.claude/rules/typed-boundaries.md index e95f9f3..2a968ca 100644 --- a/sdk/python/.claude/rules/typed-boundaries.md +++ b/sdk/python/.claude/rules/typed-boundaries.md @@ -14,17 +14,16 @@ is a ``TypedDict`` matching the server's Valibot schema: from typing import Literal, NotRequired, TypedDict class ReplayCreateBody(TypedDict): - conversationId: str - conversationVersion: str - modality: Literal["voice"] - runConfig: NotRequired[JsonObject] + conversation_hash: str + run_config: NotRequired[JsonObject] ``` -- Field names match the server (`camelCase`). The SDK speaks the - server's wire shape; we do not translate at this layer. +- Field names match the server's wire shape (`snake_case` — e.g. + ``conversation_hash``, ``run_config``). The SDK speaks the server's + shape; we do not translate at this layer. - Optional fields use ``NotRequired[...]``, not ``Required`` with a ``None`` value. -- ``runConfig`` is the *one* place ``JsonObject`` (= ``dict[str, +- ``run_config`` is the *one* place ``JsonObject`` (= ``dict[str, JsonValue]``) is allowed — it's an opaque pass-through the dev owns. Use ``xray._json.JsonObject``, never ``dict[str, Any]``. @@ -37,7 +36,10 @@ that into ``object`` and walks it: ```python class ReplayCreateResponse(TypedDict): id: str - status: Literal["running", "completed", "failed"] + lifecycle_state: Literal[ + "pending", "running", "recording_uploaded", + "analyzing", "completed", "failed", + ] raw: object = response.json() if not isinstance(raw, dict) or "id" not in raw: diff --git a/sdk/python/README.md b/sdk/python/README.md index 12c5872..1ff152f 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -1,16 +1,18 @@ # xray-py -Python SDK for [xray](https://github.com/xray-eval/xray) — replay/eval framework for LiveKit voice agents. +Python SDK for [xray](https://github.com/xray-eval/xray) — an open-source, +self-hosted replay/eval framework for LiveKit voice agents. > **Alpha.** Wire and API surface can break between minor versions. ## Install ```bash -pip install xray-py[livekit] +pip install "xray-py[livekit]" ``` -The `[livekit]` extra pulls in the `livekit` Python client. Drop it if you implement your own runtime. +The `[livekit]` extra pulls in the `livekit` client + API; drop it if you +implement your own `Runtime`. Add `[live]` for the OS-mic `run_live` session. ## Quickstart @@ -19,19 +21,14 @@ import asyncio import os from xray import Assertion, Conversation, Judge, RunConfig, Turn, format_failures, run -from xray.conversation import TtsAudio from xray.runtime.livekit import LiveKitRuntime async def main() -> None: conv = Conversation( - name="Books a table for two", + name="books-a-table-for-two", turns=[ - Turn.user( - "Hi, I'd like to book a table for two at 7pm.", - key="u0", - audio=TtsAudio(), # or RecordedAudio(path="...wav") - ), + Turn.user("Hi, I'd like to book a table for two at 7pm.", key="u0"), Turn.agent( key="a0", assertions=( @@ -63,40 +60,31 @@ async def main() -> None: asyncio.run(main()) ``` -`Conversation` identity is a SHA-256 content hash over the canonical spec (turns + per-turn `Assertion`s + conversation-level `Judge`s, with per-turn `RecordedAudio` bytes substituted in by sha256). `name` is a free-form display label — renaming the same spec attaches another Replay to the same Conversation row; editing any turn, assertion, judge, or WAV forks a new Conversation. - -The runtime produces **one stereo WAV per replay** (left = user, right = agent); `run(...)` uploads it to `POST /v1/replays/:id/audio`. The inspector slices it per-turn using the `replay_turns` timestamps. - -When a user `Turn` uses `TtsAudio()` (or has no `audio` + a text fallback), the runtime calls OpenAI's `/v1/audio/speech` directly using `OPENAI_API_KEY` from the SDK process's environment, and caches the result per-turn keyed on `(text, voice, model)` so re-runs reuse the bytes. Changing any of those invalidates the cache for that turn. - -> **Note:** the xray *server* also needs `OPENAI_API_KEY` set in its own environment — it uses the key for Whisper (per-turn transcription) and the judge LLM during `/analyze`. The SDK and server hold the same env var name; both need it for their respective stages. +Wire your agent's worker entrypoint in one block: -## Three modules - -- `xray.conversation` — `Conversation`, `Turn` test-definition primitives (`Turn.user(...)` / `Turn.agent(...)`). -- `xray.instrument` — `xray.attach(ctx, ...)` async context manager for LiveKit Agents worker entrypoints. Auto-binds the replay context from the JWT's `xray` attribute, installs the OTLP/JSON exporter, and force-flushes spans on exit. `xray.otel` exposes the lower-level `install` / `XraySpanExporter` / `XrayBaggageSpanProcessor` if you wire things manually. -- `xray.runtime` — pluggable `Runtime` ABC; `xray.runtime.livekit.LiveKitRuntime` is the v1 implementation. +```python +import xray -## Environment +async def entrypoint(ctx): + await ctx.connect() + async with xray.attach(ctx, service_name="my-agent"): + ... # your existing agent code — unchanged +``` -The LiveKit runtime reads: +`attach` reads the replay context from the joining participant's JWT `xray` +attribute, installs an OTLP/JSON exporter pointed at xray, and force-flushes +spans on exit. Note: **TTS for user turns is done server-side** — the SDK reads +no provider key and only `XRAY_OTLP_ENDPOINT` from its environment. -| Var | Required? | Default | Purpose | -|---|---|---|---| -| `LIVEKIT_URL` / `LIVEKIT_API_KEY` / `LIVEKIT_API_SECRET` | yes | — | LiveKit credentials | -| `OPENAI_API_KEY` | only for TTS turns | — | OpenAI key used directly by the SDK runtime for TTS. **xray's server also reads this var from its own env** for Whisper + judge. | -| `OPENAI_TTS_MODEL` | no | `gpt-4o-mini-tts` | TTS model | -| `OPENAI_TTS_VOICE` | no | `alloy` | Voice; per-turn `TtsAudio(voice_id=...)` overrides | +## Full reference -Recorded audio must be **48 kHz mono 16-bit WAV** (`ffmpeg -i in.wav -ar 48000 -ac 1 -sample_fmt s16 out.wav`). +This README is a quickstart. The authoritative SDK reference — every export, +signature, runtime, and error — lives in the main repo: -## How it wires to xray +- **[docs/sdk-python.md](https://github.com/xray-eval/xray/blob/main/docs/sdk-python.md)** — the SDK reference. +- [docs/integrate.md](https://github.com/xray-eval/xray/blob/main/docs/integrate.md) — end-to-end integration walkthrough. +- [docs/wire-contract.md](https://github.com/xray-eval/xray/blob/main/docs/wire-contract.md) — the OTLP attribute contract. -1. `run(...)` POSTs the Conversation to `/v1/conversations` (multipart with the canonical `spec` JSON + one file part per `RecordedAudio` turn). The server hashes the spec and upserts the conversation row by hash; assertions and judges are part of the hashed identity. It returns `{hash}`. -2. `run(...)` POSTs the Replay to `/v1/replays` referencing the hash. The server returns `{id}`. -3. The runtime joins the LiveKit room with `replay_id` + `conversation_hash` encoded as a JWT participant attribute (LiveKit `participant.attributes` ≥ v1.7). -4. `xray.attach(ctx, ...)` reads the attribute, sets OTEL baggage, and the `XrayBaggageSpanProcessor` lifts it onto every span. xray's OTLP receiver routes spans by `xray.replay.id` and persists what it recognizes (`xray.*`, OTel GenAI semconv, Langfuse). -5. `run(...)` uploads the mixdown WAV to `/v1/replays/:id/audio` and triggers `/v1/replays/:id/analyze`. The server runs VAD + Whisper transcription + per-turn metrics + every declared `Assertion` and `Judge`, then emits `evaluation_complete` over SSE. -6. `run(...)` reads the SSE event and returns `ReplayResult` with `passed` plus per-assertion / per-judge outcomes. Assertion failures don't raise — `assert result.passed` is the pytest idiom. +## License -See `docs/integrate.md` and `docs/architecture.md` in the main repo for the contract. +[Elastic License 2.0](https://github.com/xray-eval/xray/blob/main/LICENSE).