Voice clone: custom voice for outreach calls

[fyang0507]

Problem

The voice agent uses Gemini's prebuilt voices (Aoede, Puck, etc.) — generic AI voices. For outreach where the agent represents a specific person, the voice should sound like them or like a consistent branded persona.

Update 2026-05-04 — xAI Voice Agent API unblocks this

xAI shipped Grok Voice Think Fast 1.0 + Custom Voices (launched 2026-04-30). It's a voice-native realtime agent (STT+reasoning+TTS in one hop, like Gemini Live) with voice cloning built in. This collapses the prior tradeoff: previously, voice clone meant either waiting on Gemini (Option A) or eating +200–500ms via external TTS (Option B). xAI gives us clone and sub-1s latency in one product.

A new Option E is now the recommended path. See comparative study below.

Option E: xAI Voice Agent API (grok-voice-think-fast-1.0)

• Voice clone from ~120s of reference audio, ready in <2 min, two-stage passphrase verification
• 80+ preset voices, 28 languages
• Native G.711 μ-law/A-law over WebSocket — drop-in for our existing Twilio Media Streams bridge (src/daemon/mediaStreamsBridge.ts)
• Native Twilio + Vonage + LiveKit integrations; xAI ships a Twilio demo agent
• Server VAD with configurable threshold/silence (parity with our current Gemini VAD config)
• Function calling with JSON-schema tools, parallel tool calls, MCP support
• Models: grok-voice-think-fast-1.0 (flagship); grok-voice-fast-1.0 (deprecated legacy)

Comparative study

Latency

Provider	Time-to-first-audio	Steady-state turn
Gemini Live (current) — gemini-3.1-flash-live	~960ms TTFT	300–500ms
Gemini Live — gemini-2.5-flash-native-audio	similar	300–500ms
xAI Voice Agent — grok-voice-think-fast-1.0	<1s (xAI claims "5× faster than closest competitor")	not yet independently benchmarked

Both are within the natural-conversation envelope. xAI's TTFT claim is marketing; assume rough parity until we benchmark.

Pricing

Provider	Audio rate	Effective $/hour at typical load
Gemini 3.1 Flash Live	$3.00/M audio in or $0.005/min · $12.00/M audio out or $0.018/min	~$0.11/hr (per third-party benchmark @ 2.2k tok/min)
Gemini 2.5 Flash Native Audio	$3.00/M audio in · $12.00/M audio out	similar to above
xAI Voice Agent	flat $0.05/min + ~$0.005/tool call	$3.00/hr

xAI is ~27× more expensive per call hour than Gemini at conversational token loads. For low-volume outbound outreach this is small in absolute terms (a 5-min call = $0.25 vs ~$0.01), but it's the dominant tradeoff at scale. Voice cloning itself is free on both sides — xAI bundles it; Gemini doesn't offer it.

Feature parity for our use case

Capability we use	Gemini Live (current)	xAI Voice Agent
Twilio Media Streams (mulaw 8kHz WS)	✅ via our mediaStreamsBridge.ts	✅ native G.711 μ-law over WebSocket; Twilio demo exists
Concurrent sessions per daemon	✅	✅ (standard session model)
Server VAD + barge-in	✅	✅ configurable threshold/silence
System instruction (persona/objective/hangup-when)	✅	✅
end_call function tool	✅	✅ custom JSON-schema functions
send_dtmf function tool	✅ (model→tool call→Twilio REST calls.update w/ TwiML <Play digits>)	✅ same path — send_dtmf is just a function declaration, daemon-side handler in mediaStreamsBridge.ts is provider-agnostic
Voice cloning	❌ (8 prebuilt voices)	✅ (the whole point)
Transcript stream for call listen	✅	✅ (turn events on the realtime channel)

Use-case verification

The use case is supported, no architectural caveats:

1. Audio path — clean. Our bridge already speaks mulaw 8kHz over WS; xAI consumes the same. transcode.ts resampling to 16k/24k can drop or be reused for outbound.
2. System instruction — clean. systemInstruction.ts already builds a single string with phone mechanics + identity + persona + per-call params; xAI takes this verbatim.
3. Function calling — both send_dtmf and end_call are straight ports. They're regular JSON-schema function declarations; the daemon-side handlers in mediaStreamsBridge.ts (handleSendDtmf POSTs TwiML <Play digits> via Twilio REST; handleEndCall closes the session) are provider-agnostic. xAI's "DTMF SIP/WebRTC only" doc note refers to input_audio_buffer.dtmf_event_received for receiving typed DTMF from the caller — we don't use that path on Gemini either.
4. Pre-connect optimization — our current trick of pre-binding a Gemini session at dial time (rebindCallbacks()) needs an xAI equivalent. xAI docs recommend "Parallel Initialization" (WS + audio capture together) but don't expose a no-op idle-then-rebind pattern. Likely fine since xAI's TTFT claim is already <1s, but worth verifying.
5. Geographic restriction — custom voices are US-only (excluding Illinois) with a 30-voice/team quota. Fine for this project; flag for any future multi-tenant deployment.

Updated recommendation

Adopt Option E behind a config switch. Concretely:

• Add voice_agent.provider: "gemini" | "xai" to <data_repo>/outreach/config.yaml, default gemini
• New src/audio/xaiVoice.ts mirroring geminiLive.ts's shape (session, function-calling, transcripts)
• mediaStreamsBridge.ts swaps the inner client based on provider; transcode/VAD/transcript-batcher/tool-call handlers stay (the existing handleSendDtmf and handleEndCall are provider-agnostic)
• Per-call voice_id resolved from outreach/config.yaml (cloned voice ID) or default preset
• Benchmark TTFT + steady-state turn against Gemini before switching default

Original options (still valid for reference)

Option	Approach	Latency impact	Status
A: Gemini custom voice	Use Gemini's own custom voice support	None	Still not available in Live API as of 2026-05
B: External TTS	Gemini reasoning + ElevenLabs/PlayHT TTS	+200-500ms per turn	Available now
C: Full external pipeline	Replace Gemini Live entirely	+1-2s (rejected in V2 planning)	Available but too slow
D: Style transfer	Blend reference voice with preset	Minimal	Provider-dependent
E: xAI Voice Agent (NEW)	Drop-in voice-native replacement with voice clone	Claimed parity with Gemini Live	Available 2026-04-30 — recommended

Dependencies

Independent of issues #1-#6. If pursued alongside #2 (cost guardrails), coordinate on mediaStreamsBridge.ts changes. Cost guardrails matter more under Option E given the ~27× per-minute price gap with Gemini.

Details

See docs/done/voice-clone.md for the original analysis (Options A–D, latency budget for external TTS, open questions). That document predates the xAI launch and should be refreshed if Option E moves forward.