Skip to content

feat(agent): voice language switch availability and opt-in auto docs #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akirilyuk merged 2 commits into from
Sep 29, 2026
Merged

feat(agent): voice language switch availability and opt-in auto docs #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f683272
feat(agent): voice language switch availability and opt-in auto docs
akirilyuk Sep 29, 2026
2d7b28e
chore(repo): release prep 0.9.1
akirilyuk Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,17 @@ Format based on [Keep a Changelog](https://keepachangelog.com/). Versioning foll

## [Unreleased]

## [0.9.0] - 2026-09-27
## [0.9.1] - 2026-09-29

### Added

- **`setVoiceLanguage`** — agent code changes STT, TTS, or both for one live session, including the vendor (`sttVendor` / `ttsVendor`). `scope: "stt"` or `"tts"` leaves the other side in place. Spoken-language detection does not change either side until you call it. API keys stay in project secrets.
- **`language-switch` template** — `onUserLanguage` switches the Sherpa voice and STT model as two calls. Chat commands `/tts` and `/stt` change one vendor mid-conversation.
- **`getVoiceLanguage`** — last known ISO 639-1 from a successful switch ack or LID / `voice_language_changed` events.
- **`onVoiceLanguageChanged`** — `defineAgent` handler when the runner commits a language change (`voice_language_changed` IPC).
- **`voiceLanguageSwitchAvailable`** on `session_start` plus **`isVoiceLanguageSwitchAvailable`** / **`isRunnerLidAutoSwitchEnabled`** helpers for manual vs project auto-switch.
- **`language-switch` template** — `onUserLanguage` switches the Sherpa voice and STT model as two calls. Chat commands `/tts` and `/stt` change one vendor mid-conversation. When `session_start.env` has truthy **`SHERPA_LID_AUTO_SWITCH`**, the template skips `setVoiceLanguage` in `onUserLanguage` (runner owns the switch; use `onVoiceLanguageChanged` for prompts).

## [0.9.0] - 2026-09-27

## [0.8.3] - 2026-09-26

Expand Down
57 changes: 36 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,17 +180,20 @@ On plans that include project Redis, the runner injects **`AGENT_REDIS_URL`** in

For inbound HTTP webhooks, configure **`AGENT_WEBHOOK_SIGNING_SECRET`** in project settings. The runner forwards the exact request bytes on process-wide **`onWebhook`** IPC (not session-queued). Verify HMAC on `ctx.body` before `JSON.parse` — VoiceThere does not verify signatures in the SDK. See [`templates/webhooks/agent.ts`](./templates/webhooks/agent.ts).

| Export | Purpose |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `defineAgent` | Register `onAgentStart`, `onWebhook`, `onSessionStart`, `onSpeechEvent`, `onUserSpeechFinal`, `onUserLanguage`, `onSessionEnd` |
| `SpeechEvent`, `SpeechEventType` | Re-exported **types** from `@node-webrtc-rust/sdk/voice` |
| `SPEECH_EVENT_TYPE` | Import from `@node-webrtc-rust/sdk/voice` (runtime constants; not bundled into child) |
| `speak` | Request parent TTS |
| `setVoiceLanguage` | Change STT, TTS, or the vendor for one live session. `scope` selects one side. Detection does not do this until you call it. |
| `startRecording` / `pauseRecording` / `resumeRecording` / `stopRecording` | Request parent conversation recording control |
| `setConversationHistoryEnabled` / `enableConversationHistory` / `disableConversationHistory` | Stop or resume conversation history storage for one session |
| `agentLog` | Forward structured logs to parent |
| `ParentToChildMessage` / `ChildToParentMessage` | IPC contract shared with the VoiceThere agent runner |
| Export | Purpose |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `defineAgent` | Register handlers including `onUserLanguage`, `onVoiceLanguageChanged`, `onSessionStart`, … |
| `SpeechEvent`, `SpeechEventType` | Re-exported **types** from `@node-webrtc-rust/sdk/voice` (includes `voice_language_changed`, …) |
| `SPEECH_EVENT_TYPE` | Import from `@node-webrtc-rust/sdk/voice` (runtime constants; not bundled into child) |
| `speak` | Request parent TTS |
| `setVoiceLanguage` | Change STT, TTS, or the vendor for one live session (`scope`, Sherpa catalog ids, cloud vendors). |
| `getVoiceLanguage` | Last known ISO 639-1 from a successful switch ack or LID / `voice_language_changed` events. |
| `isVoiceLanguageSwitchAvailable` | `true` when `session_start.voiceLanguageSwitchAvailable` was set (runner supports agent IPC). |
| `isRunnerLidAutoSwitchEnabled` | `true` when `session_start.env.SHERPA_LID_AUTO_SWITCH` is truthy (runner auto-switch; optional in env). |
| `startRecording` / `pauseRecording` / `resumeRecording` / `stopRecording` | Request parent conversation recording control |
| `setConversationHistoryEnabled` / `enableConversationHistory` / `disableConversationHistory` | Stop or resume conversation history storage for one session |
| `agentLog` | Forward structured logs to parent |
| `ParentToChildMessage` / `ChildToParentMessage` | IPC contract shared with the VoiceThere agent runner |

### Runner runtime subpath (minimal shared sandbox API)

Expand All @@ -212,19 +215,31 @@ session orchestration and crash policy remain in the runner codebase.

Forwarded from the runner voice pipeline as SDK `SpeechEvent` payloads on `speech_event.event` (`event.type`, optional `text` / `error`):

| Event | Typical use in custom agent |
| --------------------------------------------- | ------------------------------------------------------- |
| `user_speaking_start` / `user_speaking_end` | UI state, turn-taking |
| `user_speech_partial` | Live captions, early barge-in logic |
| `user_speech_final` | Primary turn boundary (`onUserSpeechFinal` convenience) |
| `user_language` | Detected ISO 639-1 code (`onUserLanguage`). Call `setVoiceLanguage` yourself to change STT, TTS, or the vendor. See [`templates/language-switch`](./templates/language-switch/agent.ts). |
| `agent_speaking_start` / `agent_speaking_end` | Know when TTS playback starts/stops |
| `barge_in` | User interrupted agent playback |
| `vad_triggered`, `stt_stream_*`, `user_stt_*` | Low-level pipeline hooks |
| `error` | Vendor or pipeline failure |
| Event | Typical use in custom agent |
| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `user_speaking_start` / `user_speaking_end` | UI state, turn-taking |
| `user_speech_partial` | Live captions, early barge-in logic |
| `user_speech_final` | Primary turn boundary (`onUserSpeechFinal` convenience) |
| `user_language` | Detected ISO 639-1 (`onUserLanguage`). Does not change STT/TTS unless you call `setVoiceLanguage` or the project enables runner auto-switch. |
| `voice_language_switching` / `voice_language_changed` / `voice_language_switch_failed` | Runner language switch lifecycle; `onVoiceLanguageChanged` on `voice_language_changed`. |
| `agent_speaking_start` / `agent_speaking_end` | Know when TTS playback starts/stops |
| `barge_in` | User interrupted agent playback |
| `vad_triggered`, `stt_stream_*`, `user_stt_*` | Low-level pipeline hooks |
| `error` | Vendor or pipeline failure |

Copy [`templates/voice-starter/agent.ts`](./templates/voice-starter/agent.ts) as a starting point — exhaustive `switch` over speech event types with per-peer state stubs and `agentLog` tracing.

### Spoken-language switching

Sherpa **voice** and **STT** catalog ids (for example `de`, `en-lessac`, `en-small`) are documented on VoiceThere at `/docs/spoken-language`.

| Approach | Who changes STT/TTS | Agent hooks |
| ----------------------- | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Manual (default)** | Your code calls `setVoiceLanguage` per side or vendor | `onUserLanguage` → `setVoiceLanguage`; use `isVoiceLanguageSwitchAvailable` before relying on IPC |
| **Project auto-switch** | Runner when LID detects a new language (opt-in project voice setting) | `onUserLanguage` / `onVoiceLanguageChanged` for prompts; do not double-call `setVoiceLanguage` on every detection |

See [`templates/language-switch`](./templates/language-switch/README.md) for both modes.

## Multiplayer / shared state

Each end-user connection must call **`startSession()`** once — one orchestrator session id per client (one signaling room per client). Do **not** join multiple browsers to the same session credentials.
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@voicethere/agent",
"version": "0.9.0",
"version": "0.9.1",
"description": "VoiceThere customer agent SDK — IPC types and runtime helpers for sandboxed child bundles",
"type": "module",
"exports": {
Expand Down
4 changes: 4 additions & 0 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,9 @@ export {
getPlay,
isConversationHistoryAvailable,
isMixAvailable,
isVoiceLanguageSwitchAvailable,
isRunnerLidAutoSwitchEnabled,
getVoiceLanguage,
isTtsPoseAvailable,
parseChatText,
pauseRecording,
Expand Down Expand Up @@ -117,6 +120,7 @@ export {
type SpeechContext,
type SpeechEventContext,
type UserLanguageContext,
type VoiceLanguageChangedContext,
type WebhookContext,
type PlayOptions,
type AudioPosition,
Expand Down
11 changes: 10 additions & 1 deletion src/protocol.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ export interface SessionStartMessage {
* Absent on older runners or voice/data-only sessions — treat as `false`.
*/
mixAvailable?: boolean;
/**
* When `true`, the runner can apply {@link VoiceLanguageControlMessage} for this
* session (spoken-language switching via agent IPC). Absent on older runners —
* treat as `false`. Project auto-switch from LID is separate (runner-owned).
*/
voiceLanguageSwitchAvailable?: boolean;
/**
* When `true`, TTS pose / listener pose / positional panning APIs are available
* (voice or Voice+Data). Absent on data-only or older runners — treat as `false`.
Expand All @@ -105,11 +111,14 @@ export interface SessionStartMessage {
* Forwards one speech lifecycle event from the parent Sherpa/VAD/STT/TTS pipeline.
*
* The {@link SpeechEvent} shape matches `@node-webrtc-rust/sdk/voice` — see SDK docs
* for `SpeechEventType` semantics (`user_speech_final`, `user_language`, `barge_in`, etc.).
* for `SpeechEventType` semantics (`user_speech_final`, `user_language`, `barge_in`,
* `voice_language_switching`, `voice_language_changed`, `voice_language_switch_failed`, etc.).
*
* Delivered to customer code as `onSpeechEvent(ctx, message.event)`; `user_speech_final`
* also triggers the `onUserSpeechFinal` handler when `event.text` is non-empty, and
* `user_language` triggers `onUserLanguage` when `event.language` or `event.text` is set.
* `voice_language_changed` triggers the `onVoiceLanguageChanged` handler when a language
* code is present on the event.
*/
export interface SpeechEventMessage {
type: "speech_event";
Expand Down
Loading
Loading