feat(examples): latency-bench for end-to-end turn latency against your own agent

README.md

@@ -75,6 +75,10 @@ npm run demo:browser
 
 See [`examples/browser-test/README.md`](examples/browser-test/README.md).
 
+## Latency benchmark
+
+[`examples/latency-bench`](examples/latency-bench/README.md) runs counting-echo turns against your own deployed agent and writes per-turn end-to-end latency in the node-webrtc-rust benchmark format.
+
 ## Ending a session
 
 | Action                | API                                                                               | Server `end_reason`   |

examples/latency-bench/README.md

@@ -0,0 +1,124 @@
+# latency-bench
+
+Measures how long a VoiceThere voice agent takes to answer, from the client's side. It
+starts a session on your own project with `@voicethere/client/node`, speaks a counting phrase
+into the microphone track, waits for the agent to echo it, and records four timings per turn.
+The results use the same file format as the
+[node-webrtc-rust vendor benchmarks](https://github.com/akirilyuk/node-webrtc-rust/tree/main/benchmarks),
+so you can summarise and compare them with that project's tools.
+
+## What it measures
+
+A turn starts when the last audio frame of your clip has been written to the microphone track
+("speech end"). All four numbers are in milliseconds, taken on one monotonic clock on your
+machine, at the moment the voice-control message arrives.
+
+| Metric                         | Starts                                | Ends                                                                            |
+| ------------------------------ | ------------------------------------- | ------------------------------------------------------------------------------- |
+| `speech_end_to_final_ms`       | speech end                            | `user_speech_final` event with a non-empty text (the speech-to-text result)     |
+| `final_to_reply_start_ms`      | that `user_speech_final` event        | first `agent_speaking_start` event after it (the agent starts talking)          |
+| `speech_end_to_reply_start_ms` | speech end                            | the same `agent_speaking_start` event: the wait a caller feels                  |
+| `tts_first_audio_ms`           | agent called `speak()` (server clock) | first reply audio sent, as reported in `firstAudioMs` of `agent_speaking_start` |
+
+A metric is `null` when one of its events never arrived. The first turn of each conversation
+is recorded as `cold` (new session, new connections), the later ones as `warm`.
+
+VoiceThere's own end-to-end runs use the same events and the same start and end points. They
+differ in two ways: their load client produces its clip with text-to-speech, while this tool
+plays a recording you provide, and the load client stamps events with the wall clock instead
+of a monotonic one.
+
+The numbers include your network path to the VoiceThere region and the load on your machine.
+Close other heavy programs, note the region in `--region`, and use a machine that stays
+awake: if the host sleeps during a run, the tool prints `[host-suspended] result:
+inconclusive`, lists the gaps in `meta.json` and exits with code 3. Do not use such a run.
+
+## Prerequisites
+
+- A deployed VoiceThere project whose agent answers every final transcript with the same
+  words. A few lines with [`@voicethere/agent`](https://github.com/voicethere/agent) do it:
+
+  ```typescript
+  import { defineAgent, speak } from "@voicethere/agent";
+
+  defineAgent({
+    onUserSpeechFinal({ sessionId, text }) {
+      speak(sessionId, `echo: ${text}`);
+    },
+  });
+  ```
+
+  Keep the agent free of other work (no model calls, no greeting) so the numbers show the
+  platform and not your agent logic.
+
+- A **client** API key (`vthc_…`) for that project, in `VOICETHERE_CLIENT_API_KEY`. It is
+  never printed or written to the output files. See
+  [voicethere.io](https://voicethere.io) for how to create one.
+- The project id, in `VOICETHERE_PROJECT_ID` or `--project`.
+- A 16-bit PCM WAV file (mono or stereo, any sample rate) of someone saying a counting
+  phrase such as "one two three four five six seven eight nine ten", about 3 to 6 seconds.
+  The example ships no recording; record your own or generate one with a text-to-speech tool
+  whose licence you have. For example, on a machine with `espeak-ng` and `ffmpeg`:
+
+  ```bash
+  espeak-ng -v en -s 150 -w counting.wav "one two three four five six seven eight nine ten"
+  ```
+
+  The tool converts the clip to 48 kHz stereo and scales the loudest sample to -1 dBFS.
+
+- Node.js 22 or newer and `npm ci` in this repository.
+
+## Run
+
+```bash
+export VOICETHERE_CLIENT_API_KEY=vthc_...
+export VOICETHERE_PROJECT_ID=<your project id>
+
+npm run bench:latency -- \
+  --clip counting.wav \
+  --turns 10 \
+  --conversations 3 \
+  --region eu \
+  --out results/my-run
+```
+
+| Option                | Default                             | Meaning                                                                      |
+| --------------------- | ----------------------------------- | ---------------------------------------------------------------------------- |
+| `--clip <file>`       | required                            | WAV file to speak                                                            |
+| `--out <dir>`         | required                            | Output directory. The run refuses to start if it already holds a `raw.jsonl` |
+| `--turns <n>`         | `10`                                | Turns per conversation                                                       |
+| `--conversations <n>` | `1`                                 | Conversations, one after another (a new session each)                        |
+| `--region <label>`    | none                                | Free text stored in `environment.json`                                       |
+| `--project <id>`      | `VOICETHERE_PROJECT_ID`             | Project to call                                                              |
+| `--api-base <url>`    | `https://sessions.voicethere.io/v1` | Session API                                                                  |
+| `--model <label>`     | `default/default`                   | `stt/tts` label stored in each record                                        |
+| `--turn-timeout-ms`   | `30000`                             | Wait for a reply before the turn counts as failed                            |
+
+To use the example outside this repository, copy the folder and change the import in
+`src/run.ts` from `../../../src/node/index.js` to `@voicethere/client/node`.
+
+## Output
+
+- `raw.jsonl`: one record per answered turn, with `vendor: "voicethere"`, `leg: "e2e"`,
+  `transport: "webrtc"`, `phase`, `text: "counting-echo"`, `run` and the four `metrics`.
+- `meta.json`: schema version, turns and conversations, failed turns, start and end time, and
+  any host-suspend gaps.
+- `environment.json`: operating system, CPU, Node.js and client versions, region.
+
+No transcripts, session ids or keys are written.
+
+## Summarise and compare
+
+The summariser and compare tool live in the node-webrtc-rust repository:
+
+```bash
+git clone https://github.com/akirilyuk/node-webrtc-rust
+cd node-webrtc-rust
+node benchmarks/scripts/bench-vendors.mjs --summarize <your-out-dir>
+npm run bench:compare -- benchmarks/results/<published-dir>/<run> <your-out-dir>
+```
+
+How the library numbers are measured is described in
+[METHODOLOGY.md](https://github.com/akirilyuk/node-webrtc-rust/blob/main/benchmarks/METHODOLOGY.md).
+Those runs call the speech vendors directly, so they leave out WebRTC and the VoiceThere
+platform; this tool adds both.

examples/latency-bench/src/args.ts

@@ -0,0 +1,64 @@
+import { parseArgs } from "node:util";
+
+export const DEFAULT_API_BASE = "https://sessions.voicethere.io/v1";
+
+export type BenchArgs = {
+  turns: number;
+  conversations: number;
+  out: string;
+  region: string | null;
+  projectId: string;
+  apiBase: string;
+  clip: string;
+  model: string;
+  turnTimeoutMs: number;
+};
+
+function positiveInt(name: string, raw: string): number {
+  const value = Number(raw);
+  if (!Number.isInteger(value) || value < 1) {
+    throw new Error(`--${name} must be a positive integer`);
+  }
+  return value;
+}
+
+/** Parses the command line. The API key is not an argument: it is read from the environment. */
+export function parseBenchArgs(
+  argv: string[],
+  env: Record<string, string | undefined> = process.env,
+): BenchArgs {
+  const { values } = parseArgs({
+    args: argv,
+    options: {
+      turns: { type: "string", default: "10" },
+      conversations: { type: "string", default: "1" },
+      out: { type: "string" },
+      region: { type: "string" },
+      project: { type: "string" },
+      "api-base": { type: "string" },
+      clip: { type: "string" },
+      model: { type: "string", default: "default/default" },
+      "turn-timeout-ms": { type: "string", default: "30000" },
+    },
+    strict: true,
+  });
+  if (!values.out) throw new Error("--out <dir> is required");
+  if (!values.clip) {
+    throw new Error("--clip <file.wav> is required (a spoken counting phrase)");
+  }
+  const projectId = values.project ?? env.VOICETHERE_PROJECT_ID;
+  if (!projectId) {
+    throw new Error("pass --project <id> or set VOICETHERE_PROJECT_ID");
+  }
+  return {
+    turns: positiveInt("turns", values.turns!),
+    conversations: positiveInt("conversations", values.conversations!),
+    out: values.out,
+    region: values.region ?? null,
+    projectId,
+    apiBase: values["api-base"] ?? DEFAULT_API_BASE,
+    clip: values.clip,
+    model: values.model!,
+    turnTimeoutMs: positiveInt("turn-timeout-ms", values["turn-timeout-ms"]!),
+  };
+}

examples/latency-bench/src/environment.ts

@@ -0,0 +1,30 @@
+import { readFileSync } from "node:fs";
+import os from "node:os";
+
+function clientVersion(): string {
+  try {
+    const raw = readFileSync(
+      new URL("../../../package.json", import.meta.url),
+      "utf8",
+    );
+    return (JSON.parse(raw) as { version?: string }).version ?? "unknown";
+  } catch {
+    return "unknown";
+  }
+}
+
+/** Where and on what the run happened. No hostnames, addresses or user names. */
+export function collectEnvironment(
+  region: string | null,
+): Record<string, unknown> {
+  const cpus = os.cpus();
+  return {
+    os: { platform: os.platform(), release: os.release(), arch: os.arch() },
+    cpu: { model: cpus[0]?.model ?? "unknown", count: cpus.length },
+    memory_total_mb: Math.round(os.totalmem() / 1024 / 1024),
+    load_avg_1m_at_start: Number(os.loadavg()[0]!.toFixed(2)),
+    node_version: process.version,
+    voicethere_client_version: clientVersion(),
+    region,
+  };
+}

examples/latency-bench/src/metrics.test.ts

@@ -0,0 +1,95 @@
+import { describe, expect, it } from "vitest";
+
+import { computeTurnMetrics, parseSpeechEvent } from "./metrics.js";
+
+const ev = (
+  event: string,
+  atMs: number,
+  extra: { text?: string; payload?: Record<string, unknown> } = {},
+) => ({ event, atMs, ...extra });
+
+describe("computeTurnMetrics", () => {
+  it("measures the four metrics from a scripted turn", () => {
+    const metrics = computeTurnMetrics({
+      speechEndAtMs: 1000,
+      events: [
+        ev("user_speaking_start", 200),
+        ev("user_speech_final", 1400, { text: "one two three" }),
+        ev("agent_speaking_start", 1900, { payload: { firstAudioMs: 310 } }),
+        ev("agent_speaking_end", 4000),
+      ],
+    });
+    expect(metrics).toEqual({
+      speech_end_to_final_ms: 400,
+      final_to_reply_start_ms: 500,
+      speech_end_to_reply_start_ms: 900,
+      tts_first_audio_ms: 310,
+    });
+  });
+
+  it("reads first_audio_ms in snake case", () => {
+    const metrics = computeTurnMetrics({
+      speechEndAtMs: 0,
+      events: [
+        ev("user_speech_final", 10, { text: "x" }),
+        ev("agent_speaking_start", 20, { payload: { first_audio_ms: 5 } }),
+      ],
+    });
+    expect(metrics.tts_first_audio_ms).toBe(5);
+  });
+
+  it("returns null for every metric that needs a missing event", () => {
+    expect(computeTurnMetrics({ speechEndAtMs: 1000, events: [] })).toEqual({
+      speech_end_to_final_ms: null,
+      final_to_reply_start_ms: null,
+      speech_end_to_reply_start_ms: null,
+      tts_first_audio_ms: null,
+    });
+    const noReply = computeTurnMetrics({
+      speechEndAtMs: 1000,
+      events: [ev("user_speech_final", 1300, { text: "one" })],
+    });
+    expect(noReply.speech_end_to_final_ms).toBe(300);
+    expect(noReply.final_to_reply_start_ms).toBeNull();
+    expect(noReply.speech_end_to_reply_start_ms).toBeNull();
+    expect(noReply.tts_first_audio_ms).toBeNull();
+  });
+
+  it("ignores an empty final and a reply that started before the final", () => {
+    const metrics = computeTurnMetrics({
+      speechEndAtMs: 100,
+      events: [
+        ev("agent_speaking_start", 150),
+        ev("user_speech_final", 200, { text: "  " }),
+      ],
+    });
+    expect(metrics.speech_end_to_final_ms).toBeNull();
+    expect(metrics.speech_end_to_reply_start_ms).toBeNull();
+  });
+
+  it("drops a negative or non-finite firstAudioMs", () => {
+    const metrics = computeTurnMetrics({
+      speechEndAtMs: 0,
+      events: [
+        ev("user_speech_final", 10, { text: "x" }),
+        ev("agent_speaking_start", 20, { payload: { firstAudioMs: -1 } }),
+      ],
+    });
+    expect(metrics.tts_first_audio_ms).toBeNull();
+  });
+});
+
+describe("parseSpeechEvent", () => {
+  it("keeps non-envelope fields as payload and ignores other messages", () => {
+    const parsed = parseSpeechEvent(
+      { type: "speech_event", event: "agent_speaking_start", firstAudioMs: 12 },
+      5,
+    );
+    expect(parsed).toMatchObject({
+      event: "agent_speaking_start",
+      atMs: 5,
+      payload: { firstAudioMs: 12 },
+    });
+    expect(parseSpeechEvent({ type: "agent_speak", text: "x" }, 5)).toBeNull();
+  });
+});