From f6832723f6a3068b2a63fb1186e412b40d6a3773 Mon Sep 17 00:00:00 2001 From: akirilyuk Date: Tue, 29 Sep 2026 08:56:12 +0700 Subject: [PATCH 1/2] feat(agent): voice language switch availability and opt-in auto docs Expose session_start voiceLanguageSwitchAvailable, getVoiceLanguage, onVoiceLanguageChanged, and runner auto-switch helpers. Update the language-switch template and README for manual vs project auto-switch. --- README.md | 57 ++++++++++------- src/index.ts | 4 ++ src/protocol.ts | 11 +++- src/runtime.ts | 89 ++++++++++++++++++++++++-- templates/README.md | 60 +++++++++--------- templates/language-switch/README.md | 12 +++- templates/language-switch/agent.ts | 83 +++++++++++++++++++----- test/protocol.test.ts | 2 + test/runtime.test.ts | 6 ++ test/voice-language-control.test.ts | 98 ++++++++++++++++++++++++++++- 10 files changed, 347 insertions(+), 75 deletions(-) diff --git a/README.md b/README.md index 3ce8f39..a8c04e3 100644 --- a/README.md +++ b/README.md @@ -180,17 +180,20 @@ On plans that include project Redis, the runner injects **`AGENT_REDIS_URL`** in For inbound HTTP webhooks, configure **`AGENT_WEBHOOK_SIGNING_SECRET`** in project settings. The runner forwards the exact request bytes on process-wide **`onWebhook`** IPC (not session-queued). Verify HMAC on `ctx.body` before `JSON.parse` — VoiceThere does not verify signatures in the SDK. See [`templates/webhooks/agent.ts`](./templates/webhooks/agent.ts). -| Export | Purpose | -| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| `defineAgent` | Register `onAgentStart`, `onWebhook`, `onSessionStart`, `onSpeechEvent`, `onUserSpeechFinal`, `onUserLanguage`, `onSessionEnd` | -| `SpeechEvent`, `SpeechEventType` | Re-exported **types** from `@node-webrtc-rust/sdk/voice` | -| `SPEECH_EVENT_TYPE` | Import from `@node-webrtc-rust/sdk/voice` (runtime constants; not bundled into child) | -| `speak` | Request parent TTS | -| `setVoiceLanguage` | Change STT, TTS, or the vendor for one live session. `scope` selects one side. Detection does not do this until you call it. | -| `startRecording` / `pauseRecording` / `resumeRecording` / `stopRecording` | Request parent conversation recording control | -| `setConversationHistoryEnabled` / `enableConversationHistory` / `disableConversationHistory` | Stop or resume conversation history storage for one session | -| `agentLog` | Forward structured logs to parent | -| `ParentToChildMessage` / `ChildToParentMessage` | IPC contract shared with the VoiceThere agent runner | +| Export | Purpose | +| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| `defineAgent` | Register handlers including `onUserLanguage`, `onVoiceLanguageChanged`, `onSessionStart`, … | +| `SpeechEvent`, `SpeechEventType` | Re-exported **types** from `@node-webrtc-rust/sdk/voice` (includes `voice_language_changed`, …) | +| `SPEECH_EVENT_TYPE` | Import from `@node-webrtc-rust/sdk/voice` (runtime constants; not bundled into child) | +| `speak` | Request parent TTS | +| `setVoiceLanguage` | Change STT, TTS, or the vendor for one live session (`scope`, Sherpa catalog ids, cloud vendors). | +| `getVoiceLanguage` | Last known ISO 639-1 from a successful switch ack or LID / `voice_language_changed` events. | +| `isVoiceLanguageSwitchAvailable` | `true` when `session_start.voiceLanguageSwitchAvailable` was set (runner supports agent IPC). | +| `isRunnerLidAutoSwitchEnabled` | `true` when `session_start.env.SHERPA_LID_AUTO_SWITCH` is truthy (runner auto-switch; optional in env). | +| `startRecording` / `pauseRecording` / `resumeRecording` / `stopRecording` | Request parent conversation recording control | +| `setConversationHistoryEnabled` / `enableConversationHistory` / `disableConversationHistory` | Stop or resume conversation history storage for one session | +| `agentLog` | Forward structured logs to parent | +| `ParentToChildMessage` / `ChildToParentMessage` | IPC contract shared with the VoiceThere agent runner | ### Runner runtime subpath (minimal shared sandbox API) @@ -212,19 +215,31 @@ session orchestration and crash policy remain in the runner codebase. Forwarded from the runner voice pipeline as SDK `SpeechEvent` payloads on `speech_event.event` (`event.type`, optional `text` / `error`): -| Event | Typical use in custom agent | -| --------------------------------------------- | ------------------------------------------------------- | -| `user_speaking_start` / `user_speaking_end` | UI state, turn-taking | -| `user_speech_partial` | Live captions, early barge-in logic | -| `user_speech_final` | Primary turn boundary (`onUserSpeechFinal` convenience) | -| `user_language` | Detected ISO 639-1 code (`onUserLanguage`). Call `setVoiceLanguage` yourself to change STT, TTS, or the vendor. See [`templates/language-switch`](./templates/language-switch/agent.ts). | -| `agent_speaking_start` / `agent_speaking_end` | Know when TTS playback starts/stops | -| `barge_in` | User interrupted agent playback | -| `vad_triggered`, `stt_stream_*`, `user_stt_*` | Low-level pipeline hooks | -| `error` | Vendor or pipeline failure | +| Event | Typical use in custom agent | +| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `user_speaking_start` / `user_speaking_end` | UI state, turn-taking | +| `user_speech_partial` | Live captions, early barge-in logic | +| `user_speech_final` | Primary turn boundary (`onUserSpeechFinal` convenience) | +| `user_language` | Detected ISO 639-1 (`onUserLanguage`). Does not change STT/TTS unless you call `setVoiceLanguage` or the project enables runner auto-switch. | +| `voice_language_switching` / `voice_language_changed` / `voice_language_switch_failed` | Runner language switch lifecycle; `onVoiceLanguageChanged` on `voice_language_changed`. | +| `agent_speaking_start` / `agent_speaking_end` | Know when TTS playback starts/stops | +| `barge_in` | User interrupted agent playback | +| `vad_triggered`, `stt_stream_*`, `user_stt_*` | Low-level pipeline hooks | +| `error` | Vendor or pipeline failure | Copy [`templates/voice-starter/agent.ts`](./templates/voice-starter/agent.ts) as a starting point — exhaustive `switch` over speech event types with per-peer state stubs and `agentLog` tracing. +### Spoken-language switching + +Sherpa **voice** and **STT** catalog ids (for example `de`, `en-lessac`, `en-small`) are documented on VoiceThere at `/docs/spoken-language`. + +| Approach | Who changes STT/TTS | Agent hooks | +| ----------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| **Manual (default)** | Your code calls `setVoiceLanguage` per side or vendor | `onUserLanguage` → `setVoiceLanguage`; use `isVoiceLanguageSwitchAvailable` before relying on IPC | +| **Project auto-switch** | Runner when LID detects a new language (opt-in project voice setting) | `onUserLanguage` / `onVoiceLanguageChanged` for prompts; do not double-call `setVoiceLanguage` on every detection | + +See [`templates/language-switch`](./templates/language-switch/README.md) for both modes. + ## Multiplayer / shared state Each end-user connection must call **`startSession()`** once — one orchestrator session id per client (one signaling room per client). Do **not** join multiple browsers to the same session credentials. diff --git a/src/index.ts b/src/index.ts index 1cf8892..19ff08b 100644 --- a/src/index.ts +++ b/src/index.ts @@ -78,6 +78,9 @@ export { getPlay, isConversationHistoryAvailable, isMixAvailable, + isVoiceLanguageSwitchAvailable, + isRunnerLidAutoSwitchEnabled, + getVoiceLanguage, isTtsPoseAvailable, parseChatText, pauseRecording, @@ -117,6 +120,7 @@ export { type SpeechContext, type SpeechEventContext, type UserLanguageContext, + type VoiceLanguageChangedContext, type WebhookContext, type PlayOptions, type AudioPosition, diff --git a/src/protocol.ts b/src/protocol.ts index 8098829..c5207c3 100644 --- a/src/protocol.ts +++ b/src/protocol.ts @@ -94,6 +94,12 @@ export interface SessionStartMessage { * Absent on older runners or voice/data-only sessions — treat as `false`. */ mixAvailable?: boolean; + /** + * When `true`, the runner can apply {@link VoiceLanguageControlMessage} for this + * session (spoken-language switching via agent IPC). Absent on older runners — + * treat as `false`. Project auto-switch from LID is separate (runner-owned). + */ + voiceLanguageSwitchAvailable?: boolean; /** * When `true`, TTS pose / listener pose / positional panning APIs are available * (voice or Voice+Data). Absent on data-only or older runners — treat as `false`. @@ -105,11 +111,14 @@ export interface SessionStartMessage { * Forwards one speech lifecycle event from the parent Sherpa/VAD/STT/TTS pipeline. * * The {@link SpeechEvent} shape matches `@node-webrtc-rust/sdk/voice` — see SDK docs - * for `SpeechEventType` semantics (`user_speech_final`, `user_language`, `barge_in`, etc.). + * for `SpeechEventType` semantics (`user_speech_final`, `user_language`, `barge_in`, + * `voice_language_switching`, `voice_language_changed`, `voice_language_switch_failed`, etc.). * * Delivered to customer code as `onSpeechEvent(ctx, message.event)`; `user_speech_final` * also triggers the `onUserSpeechFinal` handler when `event.text` is non-empty, and * `user_language` triggers `onUserLanguage` when `event.language` or `event.text` is set. + * `voice_language_changed` triggers the `onVoiceLanguageChanged` handler when a language + * code is present on the event. */ export interface SpeechEventMessage { type: "speech_event"; diff --git a/src/runtime.ts b/src/runtime.ts index e59f86c..b1c72c4 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -56,6 +56,8 @@ export interface SessionContext { conversationHistoryAvailable: boolean; /** `true` when the runner session is Voice+Data and mix group APIs are available. */ mixAvailable: boolean; + /** `true` when runner {@link setVoiceLanguage} IPC is available for this session. */ + voiceLanguageSwitchAvailable: boolean; /** `true` when TTS pose / listener pose APIs are available (voice or Voice+Data). */ ttsPoseAvailable: boolean; } @@ -72,6 +74,13 @@ export interface UserLanguageContext { language: string; } +/** Runner committed a voice language switch (`speech.type` is `voice_language_changed`). */ +export interface VoiceLanguageChangedContext { + sessionId: string; + /** ISO 639-1 code after the runner applied STT/TTS (agent or auto-switch). */ + language: string; +} + export interface SpeechEventContext { sessionId: string; } @@ -131,6 +140,13 @@ export interface AgentHandlers { onUserSpeechFinal?: (ctx: SpeechContext) => void | Promise; /** Convenience handler — also invoked when `speech.type` is `user_language`. */ onUserLanguage?: (ctx: UserLanguageContext) => void | Promise; + /** + * Runner finished applying a language switch (`voice_language_changed`). Use with + * project auto-switch or after your own `setVoiceLanguage` calls. + */ + onVoiceLanguageChanged?: ( + ctx: VoiceLanguageChangedContext, + ) => void | Promise; /** Alias for {@link AgentHandlers.onSessionEnd}. */ onClientLeave?: (ctx: { sessionId: string }) => void | Promise; onSessionEnd?: (ctx: { sessionId: string }) => void | Promise; @@ -341,6 +357,10 @@ const recordingAvailableBySessionId = new Map(); const conversationHistoryAvailableBySessionId = new Map(); /** Cached from `session_start.mixAvailable` until `session_end`. */ const mixAvailableBySessionId = new Map(); +/** Cached from `session_start.voiceLanguageSwitchAvailable` until `session_end`. */ +const voiceLanguageSwitchAvailableBySessionId = new Map(); +/** Last known ISO 639-1 per session (setVoiceLanguage ack or speech events). */ +const lastVoiceLanguageBySessionId = new Map(); /** Cached from `session_start.ttsPoseAvailable` until `session_end`. */ const ttsPoseAvailableBySessionId = new Map(); /** Sessions that received `session_end`; `speak()` becomes a no-op when no live gen. */ @@ -591,6 +611,12 @@ function handleVoiceLanguageControlAck( if (!pending) return; clearTimeout(pending.timer); pendingVoiceLanguageAcks.delete(message.requestId); + if (message.ok && message.language?.trim()) { + lastVoiceLanguageBySessionId.set( + message.sessionId, + message.language.trim(), + ); + } pending.resolve({ ok: message.ok, reason: message.reason, @@ -912,6 +938,10 @@ async function handleParentMessage( message.sessionId, message.mixAvailable ?? false, ); + voiceLanguageSwitchAvailableBySessionId.set( + message.sessionId, + message.voiceLanguageSwitchAvailable ?? false, + ); ttsPoseAvailableBySessionId.set( message.sessionId, message.ttsPoseAvailable ?? false, @@ -929,6 +959,8 @@ async function handleParentMessage( conversationHistoryAvailable: message.conversationHistoryAvailable ?? false, mixAvailable: message.mixAvailable ?? false, + voiceLanguageSwitchAvailable: + message.voiceLanguageSwitchAvailable ?? false, ttsPoseAvailable: message.ttsPoseAvailable ?? false, }); sendParentMessage({ @@ -954,12 +986,23 @@ async function handleParentMessage( if ((message.event.type as string) === "user_language") { const language = resolveUserLanguageCode(message.event); if (language) { + lastVoiceLanguageBySessionId.set(message.sessionId, language); await handlers.onUserLanguage?.({ sessionId: message.sessionId, language, }); } } + if ((message.event.type as string) === "voice_language_changed") { + const language = resolveUserLanguageCode(message.event); + if (language) { + lastVoiceLanguageBySessionId.set(message.sessionId, language); + await handlers.onVoiceLanguageChanged?.({ + sessionId: message.sessionId, + language, + }); + } + } break; case "data_channel_message": await handlers.onDataChannelMessage?.({ @@ -989,6 +1032,8 @@ async function handleParentMessage( recordingAvailableBySessionId.delete(message.sessionId); conversationHistoryAvailableBySessionId.delete(message.sessionId); mixAvailableBySessionId.delete(message.sessionId); + voiceLanguageSwitchAvailableBySessionId.delete(message.sessionId); + lastVoiceLanguageBySessionId.delete(message.sessionId); ttsPoseAvailableBySessionId.delete(message.sessionId); await (handlers.onClientLeave ?? handlers.onSessionEnd)?.({ sessionId: message.sessionId, @@ -1255,6 +1300,8 @@ export function resetAgentIpcStateForTests(): void { recordingAvailableBySessionId.clear(); conversationHistoryAvailableBySessionId.clear(); mixAvailableBySessionId.clear(); + voiceLanguageSwitchAvailableBySessionId.clear(); + lastVoiceLanguageBySessionId.clear(); ttsPoseAvailableBySessionId.clear(); inboundQueueAuthority = null; for (const [requestId, pending] of pendingRecordingAcks) { @@ -1324,11 +1371,32 @@ export function isMixAvailable(ctx: SessionContext): boolean { return ctx.mixAvailable; } +/** True when {@link SessionStartMessage.voiceLanguageSwitchAvailable} was set for the session. */ +export function isVoiceLanguageSwitchAvailable(ctx: SessionContext): boolean { + return ctx.voiceLanguageSwitchAvailable; +} + +/** + * Last known ISO 639-1 for the session: successful {@link setVoiceLanguage} ack, + * then `user_language` / `voice_language_changed` speech events. + */ +export function getVoiceLanguage(sessionId: string): string | undefined { + return lastVoiceLanguageBySessionId.get(sessionId); +} + /** True when {@link SessionStartMessage.ttsPoseAvailable} was set for the session. */ export function isTtsPoseAvailable(ctx: SessionContext): boolean { return ctx.ttsPoseAvailable; } +/** True when `session_start.env` advertises runner LID auto-switch (if forwarded). */ +export function isRunnerLidAutoSwitchEnabled( + env: Record, +): boolean { + const raw = env.SHERPA_LID_AUTO_SWITCH?.trim().toLowerCase(); + return raw === "1" || raw === "true" || raw === "yes" || raw === "on"; +} + async function sendMixControl( action: MixControlAction, payload: Omit, @@ -1607,6 +1675,10 @@ export interface SetVoiceLanguageOptions { * `provider`: `local-sherpa`, `openai`, `elevenlabs`, `cartesia`, or `google`. */ ttsVendor?: VoiceVendorSelection; + /** + * Max wait for runner `voice_language_control_ack` (default 11 minutes for cold Sherpa pools). + */ + timeoutMs?: number; } function cleanVendor( @@ -1618,7 +1690,9 @@ function cleanVendor( provider, ...(selection?.model?.trim() ? { model: selection.model.trim() } : {}), ...(selection?.voice?.trim() ? { voice: selection.voice.trim() } : {}), - ...(selection?.language?.trim() ? { language: selection.language.trim() } : {}), + ...(selection?.language?.trim() + ? { language: selection.language.trim() } + : {}), }; } @@ -1647,10 +1721,10 @@ export function setVoiceLanguage( const ttsVendor = cleanVendor(options.ttsVendor); const hasWork = Boolean( language || - options.voice?.trim() || - options.stt?.trim() || - sttVendor || - ttsVendor, + options.voice?.trim() || + options.stt?.trim() || + sttVendor || + ttsVendor, ); if (!hasWork) { return Promise.resolve({ @@ -1663,11 +1737,14 @@ export function setVoiceLanguage( return Promise.resolve({ ok: true, reason: "local_mock", requestId }); } + const ackTimeoutMs = + options.timeoutMs ?? VOICE_LANGUAGE_CONTROL_ACK_TIMEOUT_MS; + return new Promise((resolve) => { const timer = setTimeout(() => { pendingVoiceLanguageAcks.delete(requestId); resolve({ ok: false, reason: "timeout", requestId }); - }, VOICE_LANGUAGE_CONTROL_ACK_TIMEOUT_MS); + }, ackTimeoutMs); pendingVoiceLanguageAcks.set(requestId, { resolve, timer }); sendParentMessage({ diff --git a/templates/README.md b/templates/README.md index 8858fc0..ff2e6fe 100644 --- a/templates/README.md +++ b/templates/README.md @@ -17,10 +17,10 @@ import { ## Product vs e2e -| Kind | Dashboard create | Prebuilt seed bundle | Typical consumer | -| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ----------------------- | +| Kind | Dashboard create | Prebuilt seed bundle | Typical consumer | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ----------------------- | | **product** | Yes (`echo`, `echo-dc`, `voice-starter`, `language-switch`, `world-sync`, `world-sync-binary`, `game-sync`, `voice-showcase`, `recording-consent`, `positional-tts`, `spatial-showcase`, `webhooks`, `webhooks-redis`) | Yes — `dist/templates//agent.js` | Platform project create | -| **e2e** | No | No — build from sources at test time | `voicethere/e2e` smokes | +| **e2e** | No | No — build from sources at test time | `voicethere/e2e` smokes | Product templates always set `seedOnCreate: true`. CI fails if a product template is missing its prebuilt bundle after `npm run build`. @@ -47,11 +47,11 @@ npm run build:templates Three product templates cover positional / world sync, from JSON to Redis: -| Id | Channel | State | -| -------------------- | ------------------------------- | ----------------------------- | -| `world-sync` | `onDataChannelMessage` (JSON) | One agent, in-memory, no Redis | -| `world-sync-binary` | `onDataChannelBinary` + `broadCastBinaryToClients` (`ArrayBuffer`) | One agent, in-memory, no Redis | -| `game-sync` | JSON control + binary world snapshots | Redis when `AGENT_REDIS_URL` is set | +| Id | Channel | State | +| ------------------- | ------------------------------------------------------------------ | ----------------------------------- | +| `world-sync` | `onDataChannelMessage` (JSON) | One agent, in-memory, no Redis | +| `world-sync-binary` | `onDataChannelBinary` + `broadCastBinaryToClients` (`ArrayBuffer`) | One agent, in-memory, no Redis | +| `game-sync` | JSON control + binary world snapshots | Redis when `AGENT_REDIS_URL` is set | See each folder README for the wire format. @@ -59,33 +59,33 @@ See each folder README for the wire format. Each folder has its own README. Summary: -| Id | Folder | Summary | -| -- | ------ | ------- | -| `echo` | `echo/` | Voice + chat echo | -| `echo-dc` | `echo-dc/` | Data-channel echo, no TTS | -| `voice-starter` | `voice-starter/` | Every speech event | -| `language-switch` | `language-switch/` | Separate `setVoiceLanguage` calls for STT, TTS, and vendor | -| `world-sync` | `world-sync/` | JSON pose broadcast | -| `world-sync-binary` | `world-sync-binary/` | Binary pose `ArrayBuffer` | -| `game-sync` | `game-sync/` | Authoritative sim, Redis + binary snapshots | -| `voice-showcase` | `voice-showcase/` | Conversational landing demo | -| `recording-consent` | `recording-consent/` | Recording consent flow | -| `positional-tts` | `positional-tts/` | Orbiting TTS | -| `spatial-showcase` | `spatial-showcase/` | Orbit / soundboard / proximity | -| `webhooks` | `webhooks/` | Inbound HMAC webhooks | -| `webhooks-redis` | `webhooks-redis/` | Webhooks plus Redis counter | +| Id | Folder | Summary | +| ------------------- | -------------------- | ----------------------------------------------------------------- | +| `echo` | `echo/` | Voice + chat echo | +| `echo-dc` | `echo-dc/` | Data-channel echo, no TTS | +| `voice-starter` | `voice-starter/` | Every speech event | +| `language-switch` | `language-switch/` | Manual `setVoiceLanguage` or runner auto-switch + prompt handlers | +| `world-sync` | `world-sync/` | JSON pose broadcast | +| `world-sync-binary` | `world-sync-binary/` | Binary pose `ArrayBuffer` | +| `game-sync` | `game-sync/` | Authoritative sim, Redis + binary snapshots | +| `voice-showcase` | `voice-showcase/` | Conversational landing demo | +| `recording-consent` | `recording-consent/` | Recording consent flow | +| `positional-tts` | `positional-tts/` | Orbiting TTS | +| `spatial-showcase` | `spatial-showcase/` | Orbit / soundboard / proximity | +| `webhooks` | `webhooks/` | Inbound HMAC webhooks | +| `webhooks-redis` | `webhooks-redis/` | Webhooks plus Redis counter | ## E2e templates These mirror former `e2e/fixtures/*` sources. E2E resolves entries from the package, builds into ephemeral workdirs, and uploads `dist/agent.js`. -| Id | Source | Purpose | -| ----------------- | ---------------------------------------------- | --------------------------------------------- | -| `echo-smoke` | `echo-smoke/agent.ts` | voice-smoke, agent-smoke, cli-smoke | -| `crash` | `crash/agent.ts` | session-errors-smoke, crash-policy smokes | -| `game-sync-smoke` | `game-sync-smoke/agent.ts` | deploy-smoke, shared-child, idle smokes | -| `redis-sync` | `redis-sync/agent.ts` + `world-layout.ts` | redis-sync-smoke (binary positions + Redis world blob) | -| `mix-smoke` | `mix-smoke/agent.ts` | voice-data-mix-smoke | +| Id | Source | Purpose | +| ----------------- | ----------------------------------------- | ------------------------------------------------------ | +| `echo-smoke` | `echo-smoke/agent.ts` | voice-smoke, agent-smoke, cli-smoke | +| `crash` | `crash/agent.ts` | session-errors-smoke, crash-policy smokes | +| `game-sync-smoke` | `game-sync-smoke/agent.ts` | deploy-smoke, shared-child, idle smokes | +| `redis-sync` | `redis-sync/agent.ts` + `world-layout.ts` | redis-sync-smoke (binary positions + Redis world blob) | +| `mix-smoke` | `mix-smoke/agent.ts` | voice-data-mix-smoke | **Note:** Product `echo` is not the same as e2e `echo-smoke` — keep both ids. diff --git a/templates/language-switch/README.md b/templates/language-switch/README.md index 9c0e6f9..bff8e34 100644 --- a/templates/language-switch/README.md +++ b/templates/language-switch/README.md @@ -1,12 +1,22 @@ # language-switch -Spoken-language detection tells the agent which ISO 639-1 code it heard. It does **not** change the TTS voice or the STT model. This template changes the two sides separately with `setVoiceLanguage`: +Spoken-language detection tells the agent which ISO 639-1 code it heard. It does **not** change the TTS voice or the STT model by itself. This template shows how to react in two ways: + +## Mode A — Manual switch (default) + +Leave the project **spoken-language auto-switch** setting off. When `onUserLanguage` fires, this agent calls `setVoiceLanguage` separately for each side: - `scope: "tts"` switches only the speaking voice - `scope: "stt"` switches only the listening model A Sherpa language with no STT id (`it`, `pt`, `nl`, `pl`, `hi`) fails the STT call and still switches TTS. +## Mode B — Runner auto-switch + +Enable auto-switch in the project voice settings (runner applies STT/TTS when LID detects a new language). When `session_start.env` includes a truthy `SHERPA_LID_AUTO_SWITCH`, this template **does not** call `setVoiceLanguage` from `onUserLanguage` — it logs that the runner owns the switch and updates prompts only. Use `onVoiceLanguageChanged` when you need the committed language after the runner applies the change. + +Detection and chat commands are unchanged: `/tts` and `/stt` still call `setVoiceLanguage` for one vendor at a time. + Chat commands change one vendor while the session stays connected. API keys are project secrets on the running deploy, not arguments: - `/tts sherpa de` — Sherpa TTS only diff --git a/templates/language-switch/agent.ts b/templates/language-switch/agent.ts index 12e844f..1b9377c 100644 --- a/templates/language-switch/agent.ts +++ b/templates/language-switch/agent.ts @@ -1,10 +1,14 @@ /** * Change STT and TTS separately, including the vendor, while the call stays up. * - * The runner reports `user_language` and does not change either side. This - * agent switches the Sherpa voice and the Sherpa STT model as two calls, so - * one side can fail without blocking the other. Chat commands change a single - * vendor mid-conversation: + * Two deployment modes (see README): + * + * (A) Project auto-switch off — this agent calls `setVoiceLanguage` from + * `onUserLanguage` when LID detects a new language. + * (B) Project enables runner auto-switch — STT/TTS are runner-owned; use + * `onUserLanguage` / `onVoiceLanguageChanged` for prompts only. + * + * Chat commands change a single vendor mid-conversation: * * - `/tts sherpa de` — Sherpa TTS only * - `/stt sherpa de` — Sherpa STT only @@ -17,6 +21,7 @@ import { agentLog, defineAgent, + isRunnerLidAutoSwitchEnabled, parseChatText, setVoiceLanguage, speak, @@ -39,17 +44,22 @@ const REPLIES: Record = { type SessionLanguage = { language: string; + runnerAutoSwitch: boolean; echoTimer: ReturnType | undefined; suppressNextFinal: boolean; }; const sessions = new Map(); -function stateFor(sessionId: string): SessionLanguage { +function stateFor( + sessionId: string, + env?: Record, +): SessionLanguage { const existing = sessions.get(sessionId); if (existing) return existing; const created: SessionLanguage = { language: "en", + runnerAutoSwitch: env ? isRunnerLidAutoSwitchEnabled(env) : false, echoTimer: undefined, suppressNextFinal: false, }; @@ -61,9 +71,16 @@ function replyFor(language: string): string { return REPLIES[language] ?? `Continuing in ${language}.`; } -function logSwitch(sessionId: string, side: string, result: VoiceLanguageResult): void { +function logSwitch( + sessionId: string, + side: string, + result: VoiceLanguageResult, +): void { if (!result.ok) { - agentLog("warn", `setVoiceLanguage ${side} failed: ${result.reason ?? "unknown"}`); + agentLog( + "warn", + `setVoiceLanguage ${side} failed: ${result.reason ?? "unknown"}`, + ); return; } const provider = side === "tts" ? result.ttsProvider : result.sttProvider; @@ -74,21 +91,45 @@ function logSwitch(sessionId: string, side: string, result: VoiceLanguageResult) ); } +function prepareLanguageTransition(state: SessionLanguage): void { + if (state.echoTimer) { + clearTimeout(state.echoTimer); + state.echoTimer = undefined; + } else { + state.suppressNextFinal = true; + } +} + defineAgent({ - onSessionStart({ sessionId }) { - stateFor(sessionId); - agentLog("info", `language-switch session_start ${sessionId}`); + onSessionStart({ sessionId, env }) { + const state = stateFor(sessionId, env); + if (state.runnerAutoSwitch) { + agentLog( + "info", + `language-switch session_start ${sessionId} runner LID auto-switch owns STT/TTS`, + ); + } else { + agentLog( + "info", + `language-switch session_start ${sessionId} manual setVoiceLanguage`, + ); + } }, async onUserLanguage({ sessionId, language }) { const state = stateFor(sessionId); if (!language || language === state.language) return; - if (state.echoTimer) { - clearTimeout(state.echoTimer); - state.echoTimer = undefined; - } else { - state.suppressNextFinal = true; + prepareLanguageTransition(state); + + if (state.runnerAutoSwitch) { + agentLog( + "info", + `LID detected ${language}; runner auto-switch applies STT/TTS — agent updates prompts only`, + ); + state.language = language; + speak(sessionId, replyFor(language)); + return; } const tts = await setVoiceLanguage(sessionId, { @@ -114,6 +155,18 @@ defineAgent({ speak(sessionId, replyFor(language)); }, + async onVoiceLanguageChanged({ sessionId, language }) { + const state = stateFor(sessionId); + if (!state.runnerAutoSwitch || !language || language === state.language) { + return; + } + agentLog( + "info", + `runner committed voice language ${language} for ${sessionId}`, + ); + state.language = language; + }, + async onDataChannelMessage(ctx) { const text = parseChatText(ctx.message); if (!text) return; diff --git a/test/protocol.test.ts b/test/protocol.test.ts index 858b547..cd5205a 100644 --- a/test/protocol.test.ts +++ b/test/protocol.test.ts @@ -24,10 +24,12 @@ describe("protocol", () => { env: { SESSION_ID: "peer-1", PROJECT_ID: "p", BUILD_ID: "b" }, recordingAvailable: true, mixAvailable: true, + voiceLanguageSwitchAvailable: true, }; expect(message.type).toBe("session_start"); expect(message.recordingAvailable).toBe(true); expect(message.mixAvailable).toBe(true); + expect(message.voiceLanguageSwitchAvailable).toBe(true); }); it("accepts parent recording_control_ack shape", () => { diff --git a/test/runtime.test.ts b/test/runtime.test.ts index 88f84b5..8370cd1 100644 --- a/test/runtime.test.ts +++ b/test/runtime.test.ts @@ -215,6 +215,7 @@ describe("defineAgent", () => { recordingAvailable: false, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); expect(capture.send).toHaveBeenCalledWith({ @@ -256,6 +257,7 @@ describe("defineAgent", () => { recordingAvailable: false, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); } finally { @@ -305,6 +307,7 @@ describe("defineAgent", () => { recordingAvailable: false, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); } finally { @@ -347,6 +350,7 @@ describe("defineAgent", () => { recordingAvailable: false, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); }); @@ -1852,6 +1856,7 @@ describe("session_start recordingAvailable", () => { recordingAvailable: true, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); capture.restore(); @@ -1875,6 +1880,7 @@ describe("session_start recordingAvailable", () => { recordingAvailable: false, conversationHistoryAvailable: false, mixAvailable: false, + voiceLanguageSwitchAvailable: false, ttsPoseAvailable: false, }); capture.restore(); diff --git a/test/voice-language-control.test.ts b/test/voice-language-control.test.ts index ac27307..937c552 100644 --- a/test/voice-language-control.test.ts +++ b/test/voice-language-control.test.ts @@ -1,12 +1,20 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { defineAgent, setVoiceLanguage } from "../src/runtime.js"; +import { + defineAgent, + getVoiceLanguage, + isRunnerLidAutoSwitchEnabled, + isVoiceLanguageSwitchAvailable, + resetAgentIpcStateForTests, + setVoiceLanguage, +} from "../src/runtime.js"; import { installProcessMessageCapture } from "./helpers/process-mock.js"; describe("setVoiceLanguage", () => { const childBundleEnv = process.env.__CHILD_BUNDLE_PATH__; afterEach(() => { + resetAgentIpcStateForTests(); if (childBundleEnv === undefined) { delete process.env.__CHILD_BUNDLE_PATH__; } else { @@ -117,3 +125,91 @@ describe("setVoiceLanguage", () => { capture.restore(); }); }); + +describe("voice language session helpers", () => { + afterEach(() => { + resetAgentIpcStateForTests(); + }); + + it("caches voiceLanguageSwitchAvailable on session_start", async () => { + const capture = installProcessMessageCapture(); + const onSessionStart = vi.fn(); + defineAgent({ onSessionStart }); + + capture.emit({ + type: "session_start", + sessionId: "peer-1", + env: { SESSION_ID: "peer-1" }, + voiceLanguageSwitchAvailable: true, + }); + await vi.waitFor(() => expect(onSessionStart).toHaveBeenCalled()); + expect( + isVoiceLanguageSwitchAvailable(onSessionStart.mock.calls[0][0]), + ).toBe(true); + + capture.emit({ + type: "session_start", + sessionId: "peer-2", + env: { SESSION_ID: "peer-2" }, + }); + await vi.waitFor(() => expect(onSessionStart).toHaveBeenCalledTimes(2)); + expect( + isVoiceLanguageSwitchAvailable(onSessionStart.mock.calls[1][0]), + ).toBe(false); + capture.restore(); + }); + + it("getVoiceLanguage returns language from setVoiceLanguage ack", async () => { + process.env.__CHILD_BUNDLE_PATH__ = "/tmp/agent.js"; + const capture = installProcessMessageCapture(); + defineAgent({}); + + const ackPromise = setVoiceLanguage("session-1", { language: "fr" }); + await vi.waitFor(() => expect(capture.send).toHaveBeenCalled()); + const sent = capture.send.mock.calls[0]?.[0] as { requestId: string }; + capture.emit({ + type: "voice_language_control_ack", + requestId: sent.requestId, + sessionId: "session-1", + ok: true, + reason: "applied", + language: "fr", + }); + await ackPromise; + expect(getVoiceLanguage("session-1")).toBe("fr"); + capture.restore(); + delete process.env.__CHILD_BUNDLE_PATH__; + }); + + it("dispatches onVoiceLanguageChanged from voice_language_changed speech_event", async () => { + const capture = installProcessMessageCapture(); + const onVoiceLanguageChanged = vi.fn(); + defineAgent({ onVoiceLanguageChanged }); + + capture.emit({ + type: "speech_event", + sessionId: "peer-1", + event: { type: "voice_language_changed", language: "de" }, + }); + await vi.waitFor(() => expect(onVoiceLanguageChanged).toHaveBeenCalled()); + expect(onVoiceLanguageChanged).toHaveBeenCalledWith({ + sessionId: "peer-1", + language: "de", + }); + expect(getVoiceLanguage("peer-1")).toBe("de"); + capture.restore(); + }); + + it("isRunnerLidAutoSwitchEnabled parses truthy env values", () => { + expect(isRunnerLidAutoSwitchEnabled({})).toBe(false); + expect(isRunnerLidAutoSwitchEnabled({ SHERPA_LID_AUTO_SWITCH: "1" })).toBe( + true, + ); + expect( + isRunnerLidAutoSwitchEnabled({ SHERPA_LID_AUTO_SWITCH: "true" }), + ).toBe(true); + expect(isRunnerLidAutoSwitchEnabled({ SHERPA_LID_AUTO_SWITCH: "0" })).toBe( + false, + ); + }); +}); From 2d7b28ea6c25c7801fa10f4999b0f37673b3ba91 Mon Sep 17 00:00:00 2001 From: akirilyuk Date: Tue, 29 Sep 2026 11:09:16 +0700 Subject: [PATCH 2/2] chore(repo): release prep 0.9.1 --- CHANGELOG.md | 9 +++++++-- package-lock.json | 4 ++-- package.json | 2 +- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85f67f0..d764c39 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,12 +6,17 @@ Format based on [Keep a Changelog](https://keepachangelog.com/). Versioning foll ## [Unreleased] -## [0.9.0] - 2026-09-27 +## [0.9.1] - 2026-09-29 ### Added - **`setVoiceLanguage`** — agent code changes STT, TTS, or both for one live session, including the vendor (`sttVendor` / `ttsVendor`). `scope: "stt"` or `"tts"` leaves the other side in place. Spoken-language detection does not change either side until you call it. API keys stay in project secrets. -- **`language-switch` template** — `onUserLanguage` switches the Sherpa voice and STT model as two calls. Chat commands `/tts` and `/stt` change one vendor mid-conversation. +- **`getVoiceLanguage`** — last known ISO 639-1 from a successful switch ack or LID / `voice_language_changed` events. +- **`onVoiceLanguageChanged`** — `defineAgent` handler when the runner commits a language change (`voice_language_changed` IPC). +- **`voiceLanguageSwitchAvailable`** on `session_start` plus **`isVoiceLanguageSwitchAvailable`** / **`isRunnerLidAutoSwitchEnabled`** helpers for manual vs project auto-switch. +- **`language-switch` template** — `onUserLanguage` switches the Sherpa voice and STT model as two calls. Chat commands `/tts` and `/stt` change one vendor mid-conversation. When `session_start.env` has truthy **`SHERPA_LID_AUTO_SWITCH`**, the template skips `setVoiceLanguage` in `onUserLanguage` (runner owns the switch; use `onVoiceLanguageChanged` for prompts). + +## [0.9.0] - 2026-09-27 ## [0.8.3] - 2026-09-26 diff --git a/package-lock.json b/package-lock.json index 0f8e3c5..715fbe8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@voicethere/agent", - "version": "0.9.0", + "version": "0.9.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@voicethere/agent", - "version": "0.9.0", + "version": "0.9.1", "license": "MIT", "dependencies": { "esbuild": "0.25.12" diff --git a/package.json b/package.json index ee04bc7..bc03ebd 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@voicethere/agent", - "version": "0.9.0", + "version": "0.9.1", "description": "VoiceThere customer agent SDK — IPC types and runtime helpers for sandboxed child bundles", "type": "module", "exports": {