From 002c7039bde5a68c59d02526c134a812fb1801fe Mon Sep 17 00:00:00 2001 From: David Lange Date: Mon, 20 Jul 2026 11:06:53 -0400 Subject: [PATCH] Update AssemblyAI skill: Sync keyterms_prompt/timestamps, async language defaults, Voice Agent voice roster + Sessions API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sync docs since 2026-07-10 changed the API surface in a few places the skill described incorrectly, and added new endpoints: - Sync STT keyterms: the documented field is `keyterms_prompt` (with `keyterms` / `word_boost` accepted as aliases, one of the three). The skill previously asserted Sync uses `word_boost` and *not* `keyterms_prompt` — now corrected in SKILL.md and api-reference.md (config table + example). - Sync STT word timestamps are now opt-in via `config.timestamps` (default `false`); `start`/`end` are omitted from `words` unless set. Fixed the response shape/example in SKILL.md and api-reference.md. - Async `language_code` / `language_detection` no longer default to `en_us` / `false`. Omitting `language_code` auto-runs detection; the two are mutually exclusive; `language_confidence_threshold` requires detection on. Updated the param table, Language Detection section, and a new gotcha. - `remove_audio_tags` now defaults to `"all"` (annotations stripped by default; pass `null` to keep them). Documented enum + default. - Webhook auth header constraints (length/charset, mutual + webhook_url requirement) added to the webhooks section. - Param dependencies documented: redact_pii→format_text, speaker_labels→ punctuate, sentiment_analysis→punctuate, speaker_options/speakers_expected mutual exclusivity, redact_static_entities limits. - Voice Agent voice roster refreshed (July 2026): old roster (ivy/james/ tyler/… ) is now deprecated; new current roster (anna/alba/…/paul/vera, giovanni/lola/juergen/rafael/estelle). Output languages now 6 (EN/IT/ES/ DE/PT/FR). Updated voice-agents.md and examples off deprecated `ivy`. - Voice Agent stored-agent REST create `voice` is an object `{"voice_id": "..."}`, not a bare string (WS `output.voice` stays a string). Corrected api-reference.md section 17 + added a common-mistake row. - New Sync `GET /warm` connection pre-warming endpoint documented. - New Voice Agent Sessions API (GET/DELETE /v1/sessions, artifacts) documented. LLM Gateway model list verified current (no drift). Streaming voice_focus / previous_context_n_turns already accurate. --- skills/assemblyai/SKILL.md | 11 ++- skills/assemblyai/references/api-reference.md | 78 ++++++++++++++----- .../references/audio-intelligence.md | 7 +- skills/assemblyai/references/voice-agents.md | 22 +++--- 4 files changed, 81 insertions(+), 37 deletions(-) diff --git a/skills/assemblyai/SKILL.md b/skills/assemblyai/SKILL.md index fd22803..f722ab5 100644 --- a/skills/assemblyai/SKILL.md +++ b/skills/assemblyai/SKILL.md @@ -105,9 +105,9 @@ A separate synchronous endpoint for short clips — send audio in one HTTP reque - **Required header:** `X-AAI-Model` — the current quickstart uses `universal-3-5-pro`; `u3-sync-pro` (Universal-3 Pro) is also accepted (the value in the formal spec enum) - **Auth:** `Authorization: YOUR_API_KEY` (Bearer prefix optional here, unlike the async REST API; or pass `?token=YOUR_API_KEY`) - **Body:** `multipart/form-data` with an `audio` part (`Content-Type: audio/wav` or `audio/pcm`) and an optional `config` JSON part -- **`config` fields:** `prompt` (≤4096 chars), `word_boost` (string[], ≤2048 chars total — this is the documented keyterms param for Sync, *not* `keyterms_prompt`), `conversation_context` (string or string[] — prior conversation turns oldest-first, for continuity across a multi-turn conversation; oldest dropped when over the context budget), `language_code` (ISO 639-1 string or list, default `en` — steers the default prompt toward the named language(s); **ignored when a custom `prompt` is set**), and for `audio/pcm` also `sample_rate` + `channels` (required for raw PCM; WAV reads them from its header) +- **`config` fields:** `prompt` (≤4096 chars — a contextual *description* of the audio, same framing as async/streaming), `keyterms_prompt` (string[], ≤2048 chars total — the documented keyterms param for Sync; **also accepted as `keyterms` or `word_boost`, provide only one of the three**), `conversation_context` (string or string[] — prior conversation turns oldest-first, for continuity across a multi-turn conversation; oldest dropped when over the context budget), `language_code` (ISO 639-1 string or list, default `en` — steers the default prompt toward the named language(s); **ignored when a custom `prompt` is set**), `timestamps` (boolean, default `false` — see Response below), and for `audio/pcm` also `sample_rate` + `channels` (required for raw PCM; WAV reads them from its header) - **Audio limits:** 80ms–120s, ≤40MB, 16-bit only, mono/stereo (stereo down-mixed), sample rates 8000/16000/22050/24000/32000/44100/48000 Hz -- **Response:** `{ text, words[{text, start, end, confidence}], confidence, audio_duration_ms, session_id }` — word timestamps use `start`/`end` (integer milliseconds), same field names as the async API; only the clip-level `audio_duration_ms` carries the `_ms` suffix +- **Response:** `{ text, words[{text, confidence, start?, end?}], confidence, audio_duration_ms, session_id }`. **Per-word `start`/`end` timestamps are opt-in — set `config.timestamps: true`.** With the default (`false`), no timestamps are computed and `start`/`end` are omitted from each word. When enabled they are integer milliseconds (same field names as async); words that can't be aligned still omit them. Only the clip-level `audio_duration_ms` carries the `_ms` suffix - **30s per-request deadline** (504 `inference_timeout`). For audio >120s use the async REST API; for live mic audio use Streaming. ```bash @@ -136,6 +136,7 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | `prompt` is context, not instructions | Universal-3.5 Pro's `prompt` *describes* the audio (domain/scenario/details). Formatting or behavioral commands (punctuation rules, "transcribe verbatim", negative directives like "don't…") are **not supported** and are ignored — transcription behavior is managed internally | | PII audio redaction method | `override_audio_redaction_method: "silence"` replaces PII with silence instead of default beep | | Language detection | Requires minimum 15 seconds of spoken audio for reliable results | +| Async `language_code` / `language_detection` defaults | Neither defaults to a fixed value anymore. On `POST /v2/transcript`, **omitting `language_code` auto-runs language detection** (no more implicit `en_us`). `language_code` and `language_detection` are **mutually exclusive** — set `language_detection: false` only *together with* a `language_code`; disabling detection without naming a language errors. `language_confidence_threshold` can only be set when detection is on | | LLM Gateway EU region | Only Anthropic Claude and Google Gemini models available — OpenAI models are NOT supported in EU | | Disfluencies | Enable with `disfluencies: true` to keep "um"/"uh" in the transcript | | Medical Mode unsupported language | API silently skips Medical Mode and does not charge for it — check for warning in response | @@ -175,7 +176,7 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | `claude-opus-4-20250514` / `claude-sonnet-4-20250514` on LLM Gateway | **Removed June 2026.** Use Claude Opus 4.5/4.6/4.7 or Claude Sonnet 4.5/4.6 | | Uploading to `/v2/upload` with `-d`/`--data` or a JSON body | Use `--data-binary @file` (raw bytes). `-d`/JSON returns a valid `upload_url` but transcription later fails with `Transcoding failed. File type application/json` | | Using Java/Go/C# SDKs | **Discontinued.** Use Python, JS/TS, Ruby, or raw API | -| `word_boost` on the async REST API | Use `keyterms_prompt` instead. **Exception:** the Sync STT API *does* use `word_boost` (in its `config` part) — that's its documented keyterms param | +| `word_boost` on the async REST API | Use `keyterms_prompt` instead. The Sync STT API's documented keyterms field is also `keyterms_prompt` (in its `config` part) — it *accepts* `word_boost`/`keyterms` as aliases, but prefer `keyterms_prompt` and send only one of the three | | Hardcoding v2 streaming URL | v3 (`/v3/ws`) is current; v2 still works but is legacy | | Using `speech_model=u3-rt-pro` for streaming | **Removed July 2026** from the model picker and streaming spec enum — superseded by `universal-3-5-pro` (the streaming default). Set a different model only for cost tradeoffs (`universal-streaming-english`/`-multilingual`) | | Python SDK rejects `universal-3-5-pro` | Upgrade to `assemblyai>=0.64.21` for Streaming v3 SDK support. Older SDKs such as `0.64.4` validate `speech_model` against an enum that omits `universal-3-5-pro` | @@ -183,6 +184,8 @@ See `references/llm-gateway.md` for models, tool calling, structured outputs, an | S2S `session.update` without `"session"` key | Must wrap config: `{"type":"session.update","session":{...}}` | | S2S tool schema using `{"function":{...}}` nesting | S2S tools are flat: `{"type":"function","name":"...","description":"...","parameters":{...}}` | | Voice Agent S2S URL | Correct URL: `wss://agents.assemblyai.com/v1/ws` — not `/v1/voice` (renamed April 2026), `/v1/realtime` (older), or `speech-to-speech.us.assemblyai.com` (very old) | +| Voice Agent REST create `"voice": "anna"` (string) | The stored-agent REST create/update body wants an **object**: `"voice": {"voice_id": "anna"}`. Only the WebSocket `session.output.voice` takes a plain string | +| Voice Agent default voice `ivy` (and `james`, `tyler`, `winter`, `bella`, `david`, `arjun`, …) | **Deprecated July 2026** (still functional, will be removed later). Pick a current voice — English: `anna`/`alba`/`charles`/`eve`/`george`/`jane`/`jean`/`mary`/`michael` (🇺🇸), `paul`/`vera` (🇬🇧); native-accent: `giovanni` (IT), `lola` (ES), `juergen` (DE), `rafael` (PT), `estelle` (FR). See `references/voice-agents.md` | | Voice Agent `tool.call` `args` field | Renamed to `arguments` — `event["arguments"]` is the parameter dict | | Medical Mode `domain: "medical"` | Correct value is `domain: "medical-v1"` | | LLM Gateway tool result `role: "function_call_output"` | Correct role is `"tool"` — use `{"role": "tool", "tool_call_id": "...", "content": "..."}` | @@ -204,7 +207,7 @@ Read the relevant reference file based on what the user needs: | `references/llm-gateway.md` | Applying LLMs to transcripts, tool calling, available models | | `references/speech-understanding.md` | Translation, speaker identification, custom formatting, summarization, action items | | `references/audio-intelligence.md` | PII redaction, diarization, summarization, sentiment, chapters | -| `references/api-reference.md` | Full parameter list, export endpoints, webhooks, upload, PII policies, Sync STT API, Voice Agents REST API (stored agents, HTTP-tool headers, BYO LLM, webhook subscriptions) | +| `references/api-reference.md` | Full parameter list, export endpoints, webhooks, upload, PII policies, Sync STT API (incl. `/warm` pre-warming), Voice Agents REST API (stored agents, HTTP-tool headers, BYO LLM, webhook subscriptions, Sessions API for recordings/transcripts) | ## API Spec Source of Truth diff --git a/skills/assemblyai/references/api-reference.md b/skills/assemblyai/references/api-reference.md index 3d95986..39438b2 100644 --- a/skills/assemblyai/references/api-reference.md +++ b/skills/assemblyai/references/api-reference.md @@ -35,12 +35,12 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | `speech_models` | array | **Optional.** Priority-ordered list of speech models. First supported model is used; falls back to the next. **If omitted, defaults to `["universal-3-5-pro", "universal-2"]`.** The enum now accepts only `universal-3-5-pro` and `universal-2` — `universal-3-pro` was removed (superseded by `universal-3-5-pro`). The `speech_model_used` response field reports which model actually ran. | | `prompt` | string | For Universal-3.5 Pro, a contextual *description* of the audio (domain → scenario → full detail), **not** formatting/behavioral instructions (those are ignored). **Complementary with `keyterms_prompt`** — both can be set together. | | `keyterms_prompt` | array | List of key terms/phrases (strings) to boost recognition accuracy — up to **1000** terms for Universal-3.5 Pro, **200** for Universal-2, max **6 words per phrase**. **Complementary with `prompt`** — both can be set together. | -| `language_code` | string | Language code (e.g., `"en_us"`, `"es"`, `"fr"`). Defaults to `"en_us"`. | -| `language_detection` | boolean | Enable automatic language detection. Default `false`. | +| `language_code` | string | Language code (e.g., `"en_us"`, `"es"`, `"fr"`). **No fixed default** — if you don't specify a language, it's detected automatically. **Cannot be used together with `language_detection`** (mutually exclusive). | +| `language_detection` | boolean | Enable automatic language detection. **Applied automatically when you omit `language_code`.** Set to `false` only *together with* a `language_code`; disabling it without specifying a language returns an error. | | `language_detection_options` | object | Options for language detection: `expected_languages` (array of language codes), `fallback_language` (string), `code_switching` (boolean, Universal-2 only), `code_switching_confidence_threshold` (float, default 0.3). | -| `language_confidence_threshold` | float | Minimum confidence threshold for language detection (0-1). | -| `speaker_labels` | boolean | Enable speaker diarization. Default `false`. | -| `sentiment_analysis` | boolean | Enable sentiment analysis on each sentence. Default `false`. | +| `language_confidence_threshold` | float | Minimum confidence threshold for language detection (0-1), defaults to `0`. **Can only be set when `language_detection` is enabled.** | +| `speaker_labels` | boolean | Enable speaker diarization. Default `false`. **Requires `punctuate: true`** (default). | +| `sentiment_analysis` | boolean | Enable sentiment analysis on each sentence. Default `false`. **Requires `punctuate: true`** (default). | | `entity_detection` | boolean | Enable entity detection. Default `false`. | | `auto_chapters` | boolean | **Deprecated.** Use Speech Understanding `summarization` (chaptered summary; see `speech-understanding.md`) or the LLM Gateway instead. | | `iab_categories` | boolean | Enable IAB content category detection. Default `false`. | @@ -49,14 +49,14 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | `summarization` | boolean | **Deprecated.** Use Speech Understanding `summarization`/`action_items` (see `speech-understanding.md`) or the LLM Gateway instead. | | `summary_model` | string | **Deprecated.** Model for summarization. | | `summary_type` | string | **Deprecated.** Type of summary. | -| `redact_pii` | boolean | Enable PII redaction. Default `false`. | +| `redact_pii` | boolean | Enable PII redaction. Default `false`. **Requires `format_text: true`** (default). | | `redact_pii_policies` | array | List of PII policies to redact (see PII Policies section). | | `redact_pii_sub` | string | Substitution type: `"hash"` (default) or `"entity_name"`. | | `redact_pii_audio` | boolean | Generate a redacted audio file. Default `false`. | | `redact_pii_audio_quality` | string | Quality of redacted audio: `"mp3"` or `"wav"`. | | `redact_pii_audio_options` | object | `override_audio_redaction_method: "silence"` replaces PII with silence instead of default beep. `return_redacted_no_speech_audio: true` also redacts non-speech segments. | | `redact_pii_return_unredacted` | boolean | When `true`, returns the original unredacted transcript alongside the redacted one in a single request. Response then includes `unredacted_text`, `unredacted_words`, and `unredacted_utterances`. Default `false`. | -| `redact_static_entities` | object | Literal find-and-replace redaction layered on top of standard PII redaction. Maps a custom label to a list of exact terms, e.g. `{"INTERNAL_TOOL": ["Bearclaw", "Cubclaw"]}`. Requires `redact_pii: true` (else 400); matched terms are also redacted in audio when `redact_pii_audio` is on. | +| `redact_static_entities` | object | Literal find-and-replace redaction layered on top of standard PII redaction. Maps a custom label to a list of exact terms, e.g. `{"INTERNAL_TOOL": ["Bearclaw", "Cubclaw"]}`. Requires `redact_pii: true` (else 400); matched terms are also redacted in audio when `redact_pii_audio` is on. **Limits:** up to 100 labels, each with up to 200 terms of ≤200 chars; a label may contain only letters, numbers, spaces, underscores, and hyphens (≤80 chars). | | `filter_profanity` | boolean | Filter profanity from transcript text. Default `false`. | | `disfluencies` | boolean | Include disfluencies (um, uh, etc.) in transcript. Default `false`. | | `multichannel` | boolean | Enable multichannel transcription. Default `false`. | @@ -66,11 +66,11 @@ Submit an audio file for transcription. Send a JSON body with the parameters bel | `webhook_auth_header_value` | string | Custom header value for webhook authentication. | | `auto_highlights` | boolean | Enable key phrase detection. Default `false`. | | `speech_understanding` | object | Enable Speech Understanding inline. Features nest under `speech_understanding.request` (the `request` wrapper is required): `translation`, `speaker_identification`, and/or `custom_formatting`. See `speech-understanding.md`. | -| `speakers_expected` | integer | Hint for number of speakers (diarization). Deprecated in favor of `speaker_options`. | -| `speaker_options` | object | Diarization options: `min_speakers_expected` (int, default 1), `max_speakers_expected` (int). | -| `temperature` | float | 0–1. Controls randomness. Universal-3.5 Pro only. | -| `domain` | string | Domain-specific model variant. `"medical-v1"` enables Medical Mode (EN, ES, DE, FR). Supported on Universal-3.5 Pro and Universal-2. | -| `remove_audio_tags` | string | Remove inline annotations from the transcript. `"all"` removes all (audio event markers and speaker cues); `"speaker"` removes only speaker cues while keeping other annotations. Universal-3.5 Pro only. | +| `speakers_expected` | integer | Hint for number of speakers (diarization). Positive integer. **Requires `speaker_labels: true`** and **cannot be used together with `speaker_options`.** Deprecated in favor of `speaker_options`. | +| `speaker_options` | object | Diarization options: `min_speakers_expected` (int, default 1), `max_speakers_expected` (int; when both set, min ≤ max). **Requires `speaker_labels: true`** and **cannot be used together with `speakers_expected`.** | +| `temperature` | float | 0–1. Controls randomness. Only takes effect on Universal-3.5 Pro. | +| `domain` | string | Domain-specific model variant. Enum: `"medical-v1"` (enables Medical Mode; EN, ES, DE, FR) or `null`. Supported on Universal-3.5 Pro and Universal-2. | +| `remove_audio_tags` | string | Remove Universal-3.5 Pro's inline annotations (audio event markers, speaker cues). Enum `"all"` / `"speaker"` / `null`. **Defaults to `"all"` — all inline annotations are removed by default.** `"speaker"` removes only speaker cues while keeping other annotations; pass `null` to keep all annotations. Only takes effect on Universal-3.5 Pro. | | `language_codes` | array | List of language codes for code-switching (must include `"en"`). Universal-3.5 Pro only. | | `audio_start_from` | integer | Start transcription from this time offset, in **milliseconds**. | | `audio_end_at` | integer | End transcription at this time offset, in **milliseconds**. | @@ -195,6 +195,10 @@ If your firewall requires IP allowlisting: Use `webhook_auth_header_name` and `webhook_auth_header_value` in the transcript request to include a custom authentication header on webhook requests. +- `webhook_auth_header_name` — 1–1000 chars, ASCII letters, numbers, hyphens, and underscores only. +- `webhook_auth_header_value` — 1–1000 chars, no carriage returns or newlines. +- Both fields require **each other** *and* `webhook_url` to also be set. + ### Metadata via Query Parameters You can append metadata as query parameters on the `webhook_url` (e.g., `https://example.com/hook?user_id=123&job_id=abc`). These are passed through on the webhook POST. @@ -258,7 +262,11 @@ Set `code_switching: true` inside `language_detection_options`, along with an op ## 12. Language Detection -Set `language_detection: true` to automatically detect the spoken language. +Automatic language detection identifies the spoken language and routes the request to the best model. **It runs automatically whenever you omit `language_code`** — you no longer have to set `language_detection: true` explicitly, and there is no implicit `en_us` default. + +- `language_code` and `language_detection` are **mutually exclusive.** Provide a `language_code` to force a language, *or* omit it to auto-detect — never both. +- To force a specific language and skip detection, set only `language_code`. Setting `language_detection: false` **without** a `language_code` returns an error. +- `language_confidence_threshold` can only be set when detection is active. ### Options @@ -413,14 +421,15 @@ POST https://sync.assemblyai.com/transcribe | Field | Type | Notes | |-------|------|-------| -| `prompt` | string | Custom instruction prepended to the model's system prompt. Max **4096** chars. Default applied when omitted. | -| `word_boost` | string[] | Keyterms that bias the decoder. Max **2048** chars total across all terms. (This is the documented keyterms param for Sync — *not* `keyterms_prompt`.) | +| `prompt` | string | A contextual **description** of the audio (domain → scenario → full detail), prepended to the model's default prompt — same framing as async/streaming, **not** formatting/behavioral instructions. Max **4096** chars. A managed default is applied when omitted. | +| `keyterms_prompt` | string[] | Keyterms that bias the decoder toward specific tokens. Max **2048** chars total across all terms. This is the documented keyterms param for Sync. **Also accepted as `keyterms` or `word_boost` — provide only one of the three.** | | `conversation_context` | string \| string[] | Prior turns from the same conversation in chronological order (oldest first, most recent last). Supplies the preceding dialogue as context for greater continuity across a multi-turn conversation (e.g. a user talking with a voice agent). A single string is treated as one turn. Oldest turns dropped first when over the context-window limit. | | `language_code` | string \| string[] | Language of the audio as an ISO 639-1 code, or a list for multilingual audio. Steers the default transcription prompt toward the named language(s). **Ignored when a custom `prompt` is set.** Default `en`. One of: en, es, de, fr, it, pt, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, ur, zh. | +| `timestamps` | boolean | Whether to compute per-word `start`/`end` timestamps. **Default `false`** — no timestamps are computed and `start`/`end` are omitted from each `words` object. Set `true` for exact per-word timings (extra latency; words that can't be aligned still omit them; accuracy best for English). | | `sample_rate` | integer | Required for `audio/pcm`. One of 8000, 16000, 22050, 24000, 32000, 44100, 48000. WAV reads it from the header. | | `channels` | integer | Required for `audio/pcm`. `1` or `2` (stereo down-mixed to mono internally). | -`prompt` and `word_boost` can both be set in the same `config` part. +`prompt` and `keyterms_prompt` can both be set in the same `config` part. ### Response @@ -428,7 +437,7 @@ POST https://sync.assemblyai.com/transcribe { "text": "Hi, I'm calling about my Best Buy order...", "words": [ - { "text": "Hi", "start": 0, "end": 200, "confidence": 0.91 } + { "text": "Hi", "confidence": 0.91, "start": 0, "end": 200 } ], "confidence": 0.87, "audio_duration_ms": 101567, @@ -436,7 +445,7 @@ POST https://sync.assemblyai.com/transcribe } ``` -Word timestamps use the fields `start` / `end` (integer **milliseconds**) — same field names as the async API. Note the clip-level `audio_duration_ms` does carry the `_ms` suffix. Include `session_id` in support requests. +**Per-word `start`/`end` timestamps are opt-in.** By default (`config.timestamps` unset or `false`) they are **omitted** from each `words` object, which carries only `text` + `confidence`. Set `config.timestamps: true` to get integer-**millisecond** `start`/`end` (same field names as the async API; words that couldn't be aligned still omit them). The clip-level `audio_duration_ms` always carries the `_ms` suffix. Include `session_id` in support requests. ### Audio Requirements @@ -473,11 +482,24 @@ curl -X POST https://sync.assemblyai.com/transcribe \ -H 'Authorization: YOUR_API_KEY' \ -H 'X-AAI-Model: universal-3-5-pro' \ -F 'audio=@sample.wav;type=audio/wav' \ - -F 'config={"prompt":"Transcribe verbatim.","word_boost":["AssemblyAI"]};type=application/json' + -F 'config={"prompt":"Best Buy order-status support call.","keyterms_prompt":["AssemblyAI"],"timestamps":true};type=application/json' ``` When to use: short pre-recorded clips needing an immediate response (voice messages, short call recordings, externally-segmented voice-agent utterances). For audio > 120s use the async REST API; for live mic audio use Streaming. +### Connection Pre-Warming (`GET /warm`) + +**`GET https://sync.assemblyai.com/warm`** establishes the HTTPS connection (DNS + TCP + TLS handshake) *ahead* of a `/transcribe` request so the transcription call starts uploading audio immediately instead of paying for connection setup first. + +- **Unauthenticated**, idempotent, and safe to call repeatedly — the response body carries no information; the value is the open connection left in your client's pool. +- **Required header:** `X-AAI-Model` (use the **same value** as your `/transcribe` request, e.g. `universal-3-5-pro`, so the warmed connection matches the routing path). +- The warmed connection is reused **only** when `/transcribe` goes through the **same HTTP client (connection pool)** and the **same base URL**. Idle connections are evicted after a few seconds to minutes, so warm **right before** transcribing (typically the moment recording starts), not at startup. + +```bash +curl -s https://sync.assemblyai.com/warm -H 'X-AAI-Model: universal-3-5-pro' +# then POST /transcribe on the same client/session to reuse the warmed connection +``` + ## 17. Voice Agents REST API (Stored Agents) A REST API for creating **reusable** voice agents. An agent stores its `system_prompt`, `greeting`, `voice`, `tools`, `input`, and `output` server-side; you then bind a WebSocket session to it by sending `{"agent_id": ""}` as the only field in your first `session.update` (see `references/voice-agents.md`). The same stored agent can be reused across the WebSocket API, browser, or Twilio. @@ -493,13 +515,13 @@ A REST API for creating **reusable** voice agents. An agent stores its `system_p | `PUT /v1/agents/{agent_id}` | Update an agent. Every field optional — send only what you want to change. | | `DELETE /v1/agents/{agent_id}` | Delete an agent. Returns `204`, no body. | -**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`, `llm`. Note `voice` is a **top-level** field here (in the WebSocket `session.update` it lives under `output.voice`). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first. +**Create body** (`application/json`): required `name`, `system_prompt`, `voice`; optional `greeting`, `input`, `output`, `tools`, `llm`. Note `voice` is a **top-level** field here and takes an **object** — `{"voice_id": ""}`, **not** a bare string (in the WebSocket `session.update` `output.voice` *is* a plain string). Pick a current voice id — see the voice roster in `references/voice-agents.md` (`ivy` and the older names are deprecated). `greeting` is spoken straight to TTS on connect — omit it to have the agent listen first. ```bash curl -X POST https://agents.assemblyai.com/v1/agents \ -H 'Authorization: YOUR_API_KEY' \ -H 'Content-Type: application/json' \ - -d '{"name":"Support Bot","system_prompt":"You are a concise support agent.","voice":"ivy","greeting":"Hi, how can I help?"}' + -d '{"name":"Support Bot","system_prompt":"You are a concise support agent.","voice":{"voice_id":"anna"},"greeting":"Hi, how can I help?"}' ``` ### Server-side HTTP tools (`headers` shape) @@ -541,3 +563,17 @@ A REST API (base URL `https://agents.assemblyai.com`, same auth) to subscribe UR - **Events:** `session.started`, `session.completed`, `call.connected`, `call.ended`, `call.failed`. - **Body fields:** `url` (`https` + public host, required), `events` (array, required), `secret` (32–256 printable ASCII chars, no whitespace — signing secret, **write-only**, only its `secret_version` is returned), `agent_id` (optional; omit for account-wide). + +## 19. Voice Agent Sessions API (Recordings & Transcripts) + +A REST API (base URL `https://agents.assemblyai.com`, same auth) to list, retrieve, and delete Voice Agent **sessions** and download each session's artifacts (recording, conversation timeline, metadata). + +| Method & Path | Description | +|---------------|-------------| +| `GET /v1/sessions` | List sessions, newest first. Query params: `limit` (1–200, default 50), `cursor` (from the previous response's `response_metadata.next_cursor`), `status`, `agent_id`. | +| `GET /v1/sessions/{session_id}` | Retrieve one session, including its artifacts as short-lived **pre-signed download URLs**. | +| `DELETE /v1/sessions/{session_id}` | Soft-delete a session. Returns `204`, no body. | + +**Session record fields:** `id` (`sess_...`), `agent_id` (null when no stored agent was used), `status` (e.g. `completed`), `public_close_reason`, `duration_seconds` (null while active), `config` (session config captured at start), `created_at`, `ended_at`, and `artifacts` (empty until the session completes). + +**Artifacts** (`artifacts[]`): each has `type` (`audio` | `timeline` | `metadata`), `url` (pre-signed, short TTL — re-fetch the session for a fresh link), and `content_type` (e.g. `audio/ogg`). The `timeline` artifact holds the conversation turns and config changes; guard for missing keys (`turns`/`config_changes` are omitted when empty). diff --git a/skills/assemblyai/references/audio-intelligence.md b/skills/assemblyai/references/audio-intelligence.md index 1254753..aa4ecab 100644 --- a/skills/assemblyai/references/audio-intelligence.md +++ b/skills/assemblyai/references/audio-intelligence.md @@ -4,13 +4,14 @@ All Audio Intelligence features are enabled via boolean parameters on the `POST ## Speaker Diarization -- Enable with `speaker_labels: true` +- Enable with `speaker_labels: true` (**requires `punctuate: true`**, which is the default) - Response includes `utterances` array with `speaker`, `text`, `start`, `end` - Each word also gets a `speaker` field +- Set a speaker range with `speaker_options` (`min_speakers_expected`/`max_speakers_expected`) — **requires `speaker_labels: true`** and is **mutually exclusive with the older `speakers_expected`** hint ## PII Redaction -- Enable with `redact_pii: true` +- Enable with `redact_pii: true` (**requires `format_text: true`**, which is the default) - `redact_pii_policies`: array of policy strings. Common policies include: - `person_name` - `phone_number` @@ -32,7 +33,7 @@ All Audio Intelligence features are enabled via boolean parameters on the `POST ## Sentiment Analysis -- Enable with `sentiment_analysis: true` +- Enable with `sentiment_analysis: true` (**requires `punctuate: true`**, which is the default) - Response includes `sentiment_analysis_results` array - Each result has `text`, `sentiment` (POSITIVE/NEGATIVE/NEUTRAL), `confidence`, `speaker` diff --git a/skills/assemblyai/references/voice-agents.md b/skills/assemblyai/references/voice-agents.md index 0364a79..2a8e11e 100644 --- a/skills/assemblyai/references/voice-agents.md +++ b/skills/assemblyai/references/voice-agents.md @@ -92,7 +92,7 @@ Sessions are preserved for **30 seconds** after disconnection. Reconnect using ` } }, "output": { - "voice": "ivy", + "voice": "anna", "format": { "encoding": "audio/pcm" }, "volume": 100 }, @@ -264,20 +264,24 @@ For browser apps, enable echo cancellation via `getUserMedia({ audio: { echoCanc ### Available Voices -Set a voice via `session.output.voice` in `session.update` **before `session.ready`**. `output.voice` and `output.format` are immutable once the session is established — the voice **cannot be changed mid-conversation**. (`output.volume` is the exception — it remains mutable.) Default is `ivy`. +Set a voice via `session.output.voice` in `session.update` **before `session.ready`** — over the WebSocket `output.voice` is a **plain string** (e.g. `"anna"`). `output.voice` and `output.format` are immutable once the session is established — the voice **cannot be changed mid-conversation**. (`output.volume` is the exception — it remains mutable.) On the **stored-agent REST API** the same choice is instead an **object**: `"voice": {"voice_id": "anna"}`. The default is `ivy`, which is now **deprecated** — set a current voice explicitly. -**Every voice speaks all output languages** — 🇺🇸 English, French, German, Italian, Portuguese, Spanish, Hindi, Mandarin, Russian, Korean, and Japanese. The two groups below differ only by the voice's *primary accent*, not which languages it can speak. Officially supported **output** languages (those with at least one native/primary-accent voice) are English, French, Italian, Spanish, Hindi, and Russian; German, Portuguese, Turkish, Dutch, and Swedish are on the roadmap. **Input** recognition uses Universal-3.5 Pro Streaming and covers **18 languages** with native code-switching (en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh) — an agent can speak an output language it can't transcribe (useful for translation-style flows). +The voice roster was refreshed in **July 2026**. Prefer the current roster below; the older names still work but are **deprecated and will be removed in a future release** — migrate when convenient. -**American-English accent** (carried into other languages): -`ivy`, `james`, `tyler`, `winter`, `bella`, `david`, `kyle`, `helen`, `martha`, `river`, `emma`, `victor`, `eleanor` +**Every voice speaks all supported output languages** — the two current groups below differ only by the voice's *primary accent*, not which languages it can speak. Officially supported **output** languages (each backed by ≥1 native/primary-accent voice) are **English, Italian, Spanish, German, Portuguese, and French** (6); Hindi and others are [on the roadmap](https://www.assemblyai.com/docs/voice-agents/voice-agent-api/supported-languages). **Input** recognition uses Universal-3.5 Pro Streaming and covers **18 languages** with native code-switching (en, es, de, fr, pt, it, tr, nl, sv, no, da, fi, hi, vi, ar, he, ja, zh) — an agent can speak an output language it can't transcribe (useful for translation-style flows). -**Native non-English accent** (code-switches with English): -`arjun` (Hindi/Hinglish), `dmitri` (Russian), `pierre` (French), `giulia`/`luca` (Italian), `lucia`/`mateo` (Spanish), `diego` (Latin American / Colombian Spanish) +**Current — English** (🇺🇸 American accent unless noted): +`alba`, `anna`, `charles`, `eve`, `george`, `jane`, `jean`, `mary`, `michael`; `paul`, `vera` (🇬🇧 British) -**Removed June 2026** (now rejected at `session.update`): `sam`, `mia`, `jack` (US); `sophie`, `oliver` (UK — no UK voices remain); `ethan`, `mei` (Mandarin); `lukas`, `lena` (German); `mina`, `joon` (Korean); `ren`, `hana` (Japanese). Any name not in the lists above (including older names like `autumn`, `claire`, `dawn`, `josh`, `grace`, `pete`) silently fails — call `GET https://agents.assemblyai.com/v1/voices` for the authoritative live list. +**Current — language-specific** (native accent, code-switches with English): +`giovanni` (🇮🇹 Italian), `lola` (🇪🇸 Spanish), `juergen` (🇩🇪 German), `rafael` (🇵🇹 Portuguese), `estelle` (🇫🇷 French) + +**Deprecated** (still functional; migrate off): `ivy` (the current default), `james`, `tyler`, `winter`, `bella`, `david`, `kyle`, `helen`, `martha`, `river`, `emma`, `victor`, `eleanor` (US); `arjun` (Hindi/Hinglish), `dmitri` (Russian), `pierre` (French), `giulia`/`luca` (Italian), `lucia`/`mateo` (Spanish), `diego` (Colombian Spanish) + +**Removed June 2026** (now rejected at `session.update`): `sam`, `mia`, `jack` (US); `sophie`, `oliver` (UK); `ethan`, `mei` (Mandarin); `lukas`, `lena` (German); `mina`, `joon` (Korean); `ren`, `hana` (Japanese). Any name not in the current or deprecated lists above (including older names like `autumn`, `claire`, `dawn`, `josh`, `grace`, `pete`) silently fails — call `GET https://agents.assemblyai.com/v1/voices` for the authoritative live list. ```json -{"type": "session.update", "session": {"output": {"voice": "ivy"}}} +{"type": "session.update", "session": {"output": {"voice": "anna"}}} ``` ### Voice Agent Error Codes