EchoBench

Open-source evaluation, regression testing, and observability for voice agents.

EchoBench is a full-stack benchmark harness for voice AI systems. It simulates realistic callers, conducts multi-turn conversations, injects interruptions and dependency failures, then produces evidence-backed evaluations, latency metrics, recordings, and synchronized call traces.

The project is provider-agnostic: run the deterministic local reference agent for CI, connect local models through Ollama or OpenAI-compatible endpoints, or collect metrics from hosted voice-agent platforms such as Retell.

Hosted docs: kshitijmishradev.github.io/EchoBench

Results

Local deterministic reference benchmark, run on August 4, 2026:

Metric	Result
Calls evaluated	50
Scenario mix	5 scenarios x 10 repetitions
Call pass rate	100%
Assertion pass rate	300 / 300, 100%
Median time to first audio	423 ms
p95 call time to first audio	502 ms
Worst call time to first audio	531 ms
Mean barge-in latency	124 ms
Tool-recovery calls	10 / 10 passed
Simulated call minutes	9.68 min

First hosted Retell audio smoke test:

Metric	Result
Calls evaluated	1
Retell call success	100%
Transcript assertion pass rate	75%
Median Retell E2E latency	1,070 ms
p95 Retell E2E latency	1,406 ms
ASR / LLM / TTS p50	108 ms / 657 ms / 186.5 ms
Call duration	45.9 sec

The Retell transcript miss was caused by a scenario mismatch: the call used a different customer name than the seeded standard_booking scenario expected. The collector and scenario mapping are included under tools/retell/ so future runs can be reproduced with exact call IDs and expected values.

What is included

• A FastAPI control plane and asynchronous scenario runner
• A polished React dashboard for runs, traces, scenarios, comparisons, and providers
• A versioned scenario format with caller goals, hidden behavior, and assertions
• A reference reservation agent that works without API keys
• Playable per-call audio artifacts generated locally
• Turn-level transcript, LLM, TTS, tool, and interruption events
• Deterministic assertions with expected/actual evidence
• Time-to-first-audio, p95 latency, barge-in, duration, and tool metrics
• Run-over-run comparison and regression summaries
• A provider-neutral protocol for audio and instrumentation events
• Retell call metric collection for hosted audio tests
• CLI, Docker Compose, tests, and CI

Quick start

The guided documentation site lives in docs-site/. Run make docs and open http://localhost:3000 for the quickstart, scenario format, call-trace guide, architecture, protocol, provider strategy, API reference, and troubleshooting.

Docker

docker compose up --build

Open http://localhost:3000. Click Run test suite, open a completed run, and select a call to inspect its recording and trace.

Local development

Requirements: Python 3.11+, Node 22+, uv, and FFmpeg.

make install
make api

In a second terminal:

make dashboard

Open http://localhost:5173. The Vite server proxies /api to port 8000.

CLI-only demo

make demo

The CLI runs the five seeded reservation scenarios and writes call artifacts under data/artifacts/.

Benchmark locally

Run the 50-call local reference benchmark used for the results table:

PYTHONPATH=src uv run python tools/benchmark_reference.py

The generated report is written to work/resume-metrics-reference.json. The work/ directory is intentionally ignored because it contains local reports, generated caches, and private test artifacts.

Collect Retell metrics

Create a Retell reservation agent, run web/audio calls, copy the call IDs from Retell Call History, then set your API key only in your shell:

export RETELL_API_KEY="your_retell_key"

Collect hosted call metrics:

PYTHONPATH=src uv run python tools/retell/collect.py \
  call_id_1=standard_booking \
  call_id_2=date_correction \
  call_id_3=unavailable_time

The collector writes work/retell-metrics.json and reports call success, transcript assertion pass rate, cost, duration, recordings, public logs, and latency breakdowns when Retell provides them.

How a test works

1. The scenario runner loads a caller goal, personality, scripted or generated behavior, and audio conditions.
2. A caller adapter produces a turn and a TTS adapter turns it into audio.
3. The audio channel sends frames to the agent under test and records its output.
4. Optional instrumentation events expose STT, LLM, tool, and TTS spans.
5. The evaluator applies deterministic assertions, behavioral rubrics, and audio metrics.
6. The dashboard synchronizes the recording, transcript, events, and evidence.

The built-in reference adapter keeps the development and CI loop deterministic. External agents can implement the EchoBench audio protocol and send optional trace events.

Scenario format

name: Caller changes the date
caller:
  personality: impatient
  goal: Correct the booking before it is confirmed
turns:
  - text: Book a table for two Tuesday at seven.
  - text: Wait, make that Wednesday instead.
    interrupt: true
  - text: It's Maya Shah. Go ahead.
expected:
  date: Wednesday
  time: 7:00 PM
  party_size: 2
  name: Maya Shah
conditions:
  noise: coffee_shop

The dashboard editor stores the same structure as JSON. Scenario versions are incremented on every edit so historical runs remain attributable.

EchoBench protocol

The initial audio contract uses a WebSocket carrying 20 ms binary PCM frames:

{
  "type": "session.start",
  "session_id": "call_123",
  "audio": {
    "encoding": "pcm_s16le",
    "sample_rate": 16000,
    "channels": 1,
    "frame_duration_ms": 20
  }
}

Instrumented agents may emit events like:

{
  "type": "trace.event",
  "session_id": "call_123",
  "timestamp_ms": 812,
  "name": "tool.completed",
  "data": {
    "tool": "create_reservation",
    "status": "success",
    "duration_ms": 184
  }
}

The /api/protocol/echo endpoint echoes JSON control frames and binary audio frames for adapter development.

Repository structure

src/echobench/
  api.py          HTTP, SSE, WebSocket, and artifact endpoints
  engine.py       scenario execution and reference agent
  evaluation.py   deterministic evidence-backed assertions
  protocol.py     provider-neutral event and audio models
  audio.py        local speech/tone recording artifacts
  db.py           SQLite schema and seeded reference suite
dashboard/        React and TypeScript dashboard
scenarios/        Git-friendly example scenarios
tools/retell/     Retell audio-call collection and transcript scoring
examples/         reference-agent documentation
tests/            protocol, API, evaluator, and E2E tests

Provider strategy

EchoBench distinguishes between the platform and the models it evaluates. Provider configuration and credentials live on the server, never in browser state. The seeded registry includes:

• deterministic reference caller;
• Ollama;
• OpenAI-compatible LLM endpoints;
• local system speech;
• local Whisper-compatible STT.

The reference runtime is intentionally deterministic. Production provider adapters implement the small LLM, STT, and TTS interfaces without changing scenarios or evaluation logic.

Current boundary

Version 0.1.0 is a complete, locally runnable product slice centered on the reference agent and provider-neutral contracts. It does not yet include SIP/telephony, distributed workers, hosted authentication, or a no-code agent builder. Those are deliberate post-MVP features.

Verification

make test
make lint

Verified locally:

• pytest: 8 passed
• ruff: all checks passed
• dashboard lint: passed
• dashboard production build: passed
• docs-site build/render tests: 2 passed

The backend tests execute real scenarios, create audio artifacts, and verify traces and assertions. The dashboard build runs strict TypeScript compilation.

License

MIT