Skip to content

gotchas.md

Latest commit

 

History

History
162 lines (138 loc) · 25.2 KB

gotchas.md

File metadata and controls

162 lines (138 loc) · 25.2 KB
Raw

Gotchas — UG Voice Studio

Running list of non-obvious traps, platform constraints, and corrections hit while building the Voice Studio. Append a new entry whenever a design review, verification pass, or docs check turns up something that would surprise the next engineer. Keep each entry to: the trap → the consequence → what to do, with a source anchor.

Legend: ✅ verified against installed source/repo this session · 📄 from vendor docs.

LiveKit (livekit-agents 1.5.6)

  • LiveKit's managed "Expressive mode" requires LiveKit Inference — we deliberately do NOT use it. 📄 The feature at docs.livekit.io/agents/models/tts/expressive needs an inference.TTS and only supports Fish Audio / Inworld / Cartesia / Gemini / SpaceXAI — ElevenLabs is not available, and the TTS routes through LiveKit's hosted gateway. Plan 5 hand-rolls expressive tags instead: the Databricks LLM emits them, a custom tts_text_transforms pipeline protects them, and the livekit-plugins-elevenlabs provider plugin (BYO ELEVEN_API_KEY → api.elevenlabs.io) performs them. Do not "simplify" by switching to inference.TTS — it breaks the Databricks-inference boundary and drops ElevenLabs. (Decision locked 2026-10-02; this is why the §7.8 tag pipeline exists.)
  • filter_markdown stalls streaming TTS on a bare [. ✅ The stock filter holds back the rest of the reply when it sees a [ that isn't a complete [text](url) link — so bare [whispers] would freeze audio until stream end. Run the tag-aware encode→filter_markdown→filter_emoji→decode pipeline (spec §7.8) so no bare [ reaches it. (filters.py:79-85,124-130)
  • No runtime TTS swap. ✅ AgentSession.tts is getter-only; there is no update_tts(); Agent.update_options(tts=) does not exist at 1.5.6. Change voices by overriding tts_node, never by mutating the session. (agent_session.py:1597)
  • The session closes on the 4th consecutive LLM/TTS unrecoverable error, not the 3rd. ✅ max_unrecoverable_errors=3, but the guard closes only when the count exceeds 3 (off-by-one). Keep the Halloween TTS outside the session's error count (own its failures via mark_degraded) so a vendor hiccup can't kill the call. (agent_session.py:131,1370-1406)
  • on_user_turn_completed is awaited before the reply, and an exception there drops the whole turn. ✅ The AI-Decide hook must be time-bounded (≤ cue-wait) and must never raise. turn_ctx is a per-reply temp copy: patch it for a same-turn switch, and also call update_instructions for later turns. (agent_activity.py:1964-1977,373-388)
  • Text transforms apply to the TTS branch only; the transcript branch gets raw LLM text. ✅ Tags must be stripped separately in transcription_node (and again client-side). (agent_activity.py:2407-2419,2495)

ElevenLabs / expressive-TTS

  • RESOLVED 2026-10-02 — bumped the livekit stack to 1.8.3 (commit 18c0b54). 1.8.3 is the latest version common to all five livekit packages on the Databricks mirror <internal-pypi-mirror>; pypi.org is blocked from the sandbox, and the only "1.8.4" on the mirror is the unrelated livekit-plugins-bithuman. 1.8.3 adds the Text-to-Dialogue WebSocket, so the Halloween model is eleven_v3_conversational (streams audio tags at ~280 ms — meets the ≤400 ms gate), not eleven_v4_turbo (the 1.8.3 plugin has no eleven_v4* routing; v4 would need 1.8.4+, unavailable here). UG_HALLOWEEN_TTS_MODEL default = eleven_v3_conversational.
  • The bump forced extra pins: livekit 1.1.5→1.1.18, livekit-api 1.1.0→1.2.1 (hard deps of agents 1.8.3). Re-bumping the livekit stack later is a ~7-pin edit.
  • openai SDK was downgraded 3.14→2.54 (livekit-agents 1.8.3 requires openai<3). The gateway's Responses path runs on 2.x, on local calls and on a deployed app.
  • C1–C13 anchors now sit on 1.8.3 source — the 1.5.6 line numbers in spec §4 are stale; the re-verification against 1.8.3 is in the next section.

Original 1.5.6 limitation (kept as the "why")

  • Inline tags do NOT stream at livekit-plugins-elevenlabs==1.5.6. 📄 The plugin's .stream() speaks only the multi-stream-input WebSocket, which doesn't carry the tag-performing models. eleven_v3 / eleven_v3_conversational / eleven_v4 / eleven_v4_turbo need the Text-to-Dialogue WebSocket — added in livekit-agents 1.7.1 (v3/v3_conversational, ~280 ms TTFA) and 1.8.4 (v4/v4_turbo, ~100 ms). At 1.5.6 you can only stream Flash/turbo (no tags), or run eleven_v3 over HTTP /stream per-sentence (~0.7–1.9 s TTFA — fails the ≤400 ms R8 gate). ⇒ Streamed inline tags require bumping agents+plugins to ≥1.7.1 / 1.8.4, which forces re-verifying the C1–C13 1.5.6 source anchors. (Done the same day: the stack is on 1.8.3, above.)
  • Missing ELEVEN_API_KEY raises ValueError at construction — check the env before constructing, don't rely on a lazy failure.
  • prewarm() is a no-op on the ElevenLabs plugin — the spec's "cue-time prewarm" has no effect; the WebSocket opens lazily on first .stream().
  • A profile-owned TTS is not wired to the session. The session subscribes error/metrics_collected and calls prewarm() only on its own activity.tts. For a profile-owned TTS: subscribe to its error/metrics yourself, build the StreamAdapter once, and aclose() it at shutdown (StreamAdapter.aclose() does NOT close the wrapped TTS).
  • Two SPOOKY_TAGS are undocumented (nervously, inhales deeply); ElevenLabs documents [whispers]/[laughs]/[sighs]/[exhales]/[mischievously], and v3 sometimes speaks a tag aloud. Plan 6 tested both with the configured voice and model: both were performed and neither was read aloud (docs/discovery/expressive-tags-contract.md).
  • Low R13 risk: the plugin adds no new transitive deps beyond numpy (av is already a core dep).

LiveKit 1.8.3 re-verification (supersedes the stale 1.5.6 §4 anchors)

Re-verified against installed 1.8.3 (2026-10-02). C3–C13 + text-transform/PUA behavior all HOLD (new file:lines in the SDD ledger table); only C1 & C2 changed behavior, plus a new native subsystem:

  • NEW native expressive subsystem — keep it OFF (expressive=False). 1.8.3 adds an expressive: bool|ExpressiveOptions kwarg (default False) on AgentSession/Agent/update_options that injects an LLM markup instruction, converts <expression>/<sound> XML tags → [...], and strips them from transcripts — but ONLY for providers cartesia/inworld/xai/fishaudio/gemini. ElevenLabs and OpenAI are NOT covered. ⇒ (a) our hand-rolled [whispers] tags + tts_text_transforms encode/decode + transcription_node stripping are STILL REQUIRED for ElevenLabs; (b) leave expressive=False so the framework doesn't inject a competing markup block; (c) if the Halloween voice ever becomes one of those 5 providers, prefer the native pipeline.
  • eleven_v3_conversational (a "dialogue" model) honors only stability among voice settings — similarity_boost/style/speed are dropped (plugin warns), as are chunk_length_schedule/streaming_latency/enable_ssml_parsing. ⇒ Task 5 build_tts sets only stability (= UG_HALLOWEEN_STABILITY). Routing: is_dialogue_model(m) = m.startswith("eleven_v3") → text-to-dialogue WS (tags performed + streamed). No eleven_v4* exists at 1.8.3.
  • C1 OBSOLETE: Agent.update_options(tts=...) now EXISTS (live per-agent TTS swap via _update_models). Our tts_node override stays the approach (per-utterance profile routing, no agent mutation); session-level tts is still getter-only.
  • C2 OBSOLETE: Deepgram update_options now takes model/encoding/sample_rate/bit_rate AND invalidates the pooled WS — an in-place model change now takes effect.
  • §7.8 tag filter integration: TTS-branch filtering is now a configurable text_transforms list (applied in generation.py) and plain callables are accepted — the encode/decode transforms plug in there. The AgentSession kwarg is tts_text_transforms (app/agent.py). Transcript stripping still needs transcription_node.
  • C6: preemptive_generation= is deprecated → set via turn_handling=TurnHandlingOptions(...); default behavior unchanged (ON).
  • C3 nuance: on_user_turn_completed exceptions are now caught (raise → drop turn + log, no session crash); "never raise / time-box" is still good practice but soft now.
  • The job's root span opens before the entrypoint, so the tracer provider must already exist (issue #35). ✅ 1.8.3 starts the job_entrypoint span before our entrypoint runs. A provider installed inside the entrypoint never saw that span, so MLflow, which lists only traces that have a root, showed nothing. The provider is now installed once per worker process in _setup_process (app/agent.py), with per-trace enrichment and a root-span flush in app/tracing.py. The unit tests cover the enrichment and flush logic, and the owner confirmed on a live call (2026-10-05) that traces now appear in MLflow with their root span. Tracing stays off unless DATABRICKS_TRACE_CATALOG, _SCHEMA and _TABLE_PREFIX are all set.

Databricks platform

  • No native TTS, and no realtime STT. 📄 FMAPI is chat/reasoning/embedding/image only; the AI Gateway exposes no audio paths. Batch STT exists (ai_transcribe, Beta, SQL-only) but nothing streaming. Voice I/O must be vendor-direct (Deepgram STT, ElevenLabs TTS). The only Databricks-side TTS path is self-serving an OSS model on GPU Model Serving (cold starts "one to several minutes", no documented audio streaming) — impractical for a realtime turn.
  • ai_decide REST response has no error_message field. 📄 That field belongs to the SQL VARIANT envelope, not the REST response (which documents only response + metadata.version). Detect failure as: non-2xx HTTP, null/missing response, unknown label, or confidence outside 0–1.
  • ai_decide question types are noul / choice / score — there is no bool. 📄 The probability type is noul. Never write "type":"bool" in code or the plan.
  • ai_decide is Beta, Previews-gated, and region-limited. 📄 Must be enabled per-workspace (admin → Previews). Supported AWS regions include us-east-1/-2 and us-west-2 (our workspace is us-east-1). options.version:"1.0" pins the function API, not the model — the served model can change and probability calibration can drift without a version bump, so re-check thresholds after any change.
  • No official ai_decide latency or pricing. 📄 The 0.8–2 s figure circulating is measured on TypeSafe's Jev, not Databricks-hosted ai_decide, and as of 2026-10-02 neither pricing page listed ai_decide. Measured on our workspace: 500 to 720 ms across the four calls of one tools/ai_decide_check.py run, and 583 ms for one turn on a deployed app.
  • Claude models on the gateway reject response_format: json_object (they accept json_schema only). 📄 Keep the world generator (UG_GEN_MODEL, which asks for JSON) on a GPT-family model. The generator calls /ai-gateway/openai/v1/chat/completions (src/services/uaig_chat.py), the voice LLM /responses and retrieval /embeddings; all three work against the gateway, and the generator has produced the worlds in use.

Repo (voice-agents-ug-demo)

  • src/services/gateway.post() is sync-only (blocking requests). ✅ Don't call it (or uaig_chat.complete_json) from the async turn hook — it blocks the event loop. The decider must use its own httpx.AsyncClient (httpx is already pinned).
  • _SPAN_TYPES (app/tracing.py) maps a span name to its MLflow span type, and the exporter JSON-encodes the value. ✅ To tag a span, add an entry to the map ("ug.ai_decide": "CHAIN" is there) — don't set a bare raw string.
  • The evidence router dispatches on fragment keys, not a type discriminator. ✅ handleEvidence (studio.js) checks obj.bind / obj.retrieval / obj.usage / obj.voice_mode. A new kind of payload needs its own if (obj.…) branch.
  • The transcript is stripped of expressive tags on the client too. ✅ handleTranscription (studio.js) renders stripExpressiveTags(seg.text), the belt-and-braces layer behind the server-side filter.

Worktrees & deploy

  • Apps runtime is Python 3.11; local .venv is 3.12. Recompile agent-requirements.txt for 3.11 (including the ElevenLabs plugin + its codecs extra) or the deploy build breaks.
  • The git stash stack is shared across all worktrees. Never use bare git stash / git stash pop (you could pop another session's work) — prefer a WIP commit, or git stash push -u -m "<unique-tag>" then apply <sha>.
  • New secrets aren't declared in app.yaml directly. ✅ valueFrom: points at an app resource key bound to the ug-voice-studio scope out-of-band. Adding ELEVEN_API_KEY = create the secret in the scope + attach an app resource (elevenlabs-api-key) + add the valueFrom entry in app.yaml.
  • A second app on the same LiveKit project needs its own AGENT_NAME. ✅ Workers that register under one name are candidates for the same dispatch (LiveKit's explicit-dispatch model; we never ran two under one name, since each app got its own). The web tier and the worker read the same AGENT_NAME, so set it to something unique in the deploy copy of app.yaml. diva-v1 registered as diva-v1-agent next to the older app's ug-agent, and its first call reached its own worker.
  • The worker log shows a few sub-second "event loop blocked" warnings on a first call. ✅ On 2026-10-05 a deployed app logged five, 0.15 to 0.53 s each: the first import of psycopg_pool in the job entrypoint, pydantic building the tool schemas on the first LLM turns (one of 0.53 s), and, after the call had ended, OpenTelemetry's force_flush making a synchronous OTLP export in the shutdown callback. None broke the call. Candidates if it ever matters: import psycopg_pool and build the tool schemas in prewarm, and run the flush in a thread.

Plan 5 build findings (policy / expressive / profiles)

  • The decision policy rule is a conservative safety-net; the ENGINE is the primary path. explicit_command/has_cue deliberately defer ambiguous phrasings to the engine. Accepted false-exit residuals (low-likelihood; an explicit exit rule beats the engine, so these matter — but the engine is enabled in the demo and Task 7 drops exit-rule verdicts in STANDARD): "please switch from airplane mode to normal mode" + short-gap "how do I switch the spooky voice to your regular voice" (older request-verb frame); "swap the voice plan for a regular voice plan"; "I prefer not to use the normal voice" (negation between verb + mention); a contrived 7–9-word move ("please switch the old main voice assistant on the app to regular mode"). Structural backstop: Task 7 drops any exit-rule verdict while in STANDARD (an exit is a no-op there, can only shadow an engine enter).
  • Expressive pipeline — G13 residual shapes (reach the TTS as text; low-likelihood, prompt-forbidden): nested [a [b]], a newline inside brackets, a closed group ≥119 inner chars, and an allowed tag glued to a parenthetical [sighs](softly) (parsed as a markdown link → "sighs" spoken). By design, any closed non-link [...] of 33–118 chars is DROPPED from audio AND transcript even in standard mode (a legit aside like [Note: closes at 5 PM] is lost — fine for a voice assistant). Task 11's client strip must match: closed [...] ≤120, not followed by (.
  • ElevenLabs VoiceSettings requires similarity_boost. VoiceSettings(stability=…) alone raises TypeError. Pass similarity_boost=NOT_GIVEN (the plugin drops it → only stability is sent, no "ignored option" warning). For eleven_v3* dialogue models stability is the ONLY effective setting (style/speed/similarity are ignored + warned).
  • Expressive tags are gated on the MODEL, not just the vendor. Only eleven_v3* models perform inline audio tags; flash/turbo/multilingual speak [laughs] literally. The halloween profile attaches SPOOKY_TAGS only when the model starts with eleven_v3 (mirrors the plugin's DIALOGUE_TTS_MODEL_PREFIX). If Task 1 pins a turbo model for latency, tags are automatically off (dark voice, no inline cues) — fail-closed for G13.
  • Missing UG_HALLOWEEN_VOICE_ID → Deepgram fallback (NOT ElevenLabs' stock voice). So {ELEVEN_API_KEY set, no voice_id} → the Halloween profile is the Deepgram dark voice. .env.example lists UG_HALLOWEEN_VOICE_ID and app.yaml carries it as a commented placeholder; set it in your own deploy copy.
  • build_tts can raise ValueError (missing voice id / plugin key check) or ImportError (plugin absent) → Task 8/10 wrap it and degrade. The ElevenLabs plugin calls Plugin.register_plugin on first import, which raises off the main thread — so import it at the TOP of app/agent.py (the entrypoint runs on the subprocess main thread), not lazily inside a worker thread.

Plan 5 live-run + reskin findings (2026-10-04)

  • The openai plugin does NOT auto-set reasoning effort for non-gpt-5-style model names. ✅ _supports_reasoning_effort(model) matches only bare gpt-5… names, so it returns False for system.ai.* and databricks-* — the reasoning field is never sent unless you pass it. Set reasoning=Reasoning(effort="low") (from openai.types) on openai.responses.LLM to keep server-side reasoning (and latency) in check.
  • The responses LLM's default read timeout kills long generations. ✅ A long reply ("tell me a scary story") blew past the plugin's default httpx read timeout → httpx.ReadTimeout → the turn died silently after 4 retries (the call just stops mid-conversation, no error to the user). Pass timeout=httpx.Timeout(connect=10.0, read=120.0, write=15.0, pool=10.0). Note the class is openai.responses.LLM (accepts reasoning + timeout); openai.LLM would reject reasoning.
  • use_websocket=False on the responses LLM avoids the plugin's hand-rolled WS request serializer emitting mode: null, which the gateway 400s. The HTTP path is the safe default here.
  • 1.8.3 agent.py dev removed in-process auto-reload. ✅ The worker logs "in-process auto-reload has been removed from the Python CLI; use lk agent dev for hot-reload." Editing source files does NOT reload a running worker (contrary to the older 1.5.x assumption that edits drop in-flight calls) — restart the worker to pick up code changes. The web tier serves web/public/* fresh per request, so UI edits land on the next page load with no restart.
  • CSS gradients can't be transitioned, so the Halloween cross-fade uses @property. The studio registers palette color tokens with @property and transitions THOSE on <body>; gradient-painted surfaces cross-fade for ~420ms on enter/exit (one-shot; steady-state animation is transform/opacity only; all gated under prefers-reduced-motion). Transitioning the gradients directly won't animate.

Plan 6 — expressive Halloween tags (2026-10-04)

  • The allowlist stays closed on purpose. ElevenLabs v3 accepts free-form [descriptive] tags, but nothing guarantees a given tag is performed instead of read aloud. Only tags that tools/expressive_bakeoff.py proved performed for the configured voice/model may be in src/expressive_palette.py; anything else is dropped from audio and transcript (G13). Re-run the tool when the voice, model or plugin version changes.
  • "Performed" is judged by transcript and duration, not by ear. The bake-off sees only the words Deepgram hears and the clip length, which moves in steps of about 78 ms. Its noise bar comes from one pair of plain renderings per carrier, shared by every tag: 0.16 s in the full run, barely above the 0.15 s floor. It cannot hear timbre, loudness or mood, so a tag such as whisper, soft or menacing is judged by clip length alone; 18 of the 67 tags cleared the bar on one carrier only, six cleared it with no margin, and a borderline tag can flip between runs (whispers came back no-audible-effect in a 4-synthesis smoke run). Its read-aloud check also matches whole words, so an inflected rendering ("whispered") would slip past it; none appeared in the recorded run. performed therefore means "the voice did something other than read the word", not "did the intended thing": trust spoken-aloud=0, read the rest as "no tag failed", and finish with a listen (Halloween mode: Verify the voice). (docs/discovery/expressive-tags-contract.md, Limits)
  • The ElevenLabs API key may be scoped to speech. Account and model endpoints (/v1/user/subscription, /v1/models) return 401 missing_permissions while synthesis works, so quota cannot be read and tools must never call them.
  • A .env.local makes the live Lakebase test run. tests/test_integration_data_plane.py is gated only on LAKEBASE_ENDPOINT, which app/agent.py and app/web_server.py load from .env.local at import; the test then generates two real datasets and never cleans them up. With a .env.local present, run uv run --frozen pytest -q --ignore=tests/test_integration_data_plane.py.
  • Plain uv run re-locks uv.lock to the internal proxy, and so does the commit hook. --no-sync does not stop it; --frozen does. After each git commit, git restore -- uv.lock if the hook touched it.
  • The weakest tier model follows a long prompt least reliably. On the first live run Standard (system.ai.gpt-5-nano) was the model that drifted: 90% of its cues were on the list (target 95%) and its stories ran to a median of 110 words. Premium and VIP failed differently (next two bullets). Wording fixed it, not a shorter palette: the rules now say "use only cues from the lists above" and "never end a sentence or a line with a cue", and the persona asks for a story of "four or five short sentences". Standard then scored 100% on the list, with no stage directions or stacked cues and a median story of 66 words. That pass rests on 2 samples per utterance, which is thin because several targets allow no misses (a stage direction, a stacked cue, a missing fact): re-run tools/expressive_llm_check.py with its default 3 samples (69 calls) after any change to the prompt, a tier model, the palette or the voice. (src/agent_prompt.py; contract, "LLM compliance results")
  • "Answer ONLY from the tools" makes the stronger tier models refuse to tell a story. A story is not in the tools, so Premium (system.ai.gpt-5-5) and VIP (system.ai.gpt-6-sol) answered "tell me a scary story" with a 17-24 word deflection ("I don't have a story from my tools"). It still met the density and compliance targets, with only 4-5 distinct cues, so a tier could pass on refusals. The persona now has one scoped carve-out: a spooky story is play, so the monster tells a made-up tale of four or five short sentences right away, with no tool, and keeps real orders, prices, policies and people out of it. Governance is unchanged and still the last block (G11), and business facts still come only from the tools (the "moon base" question still gets an "I don't have ..." abstention). Do not widen that line, do not drop it without re-running the LLM check, and listen to a story after any change to it. (src/agent_prompt.py, HALLOWEEN_PERSONA)
  • gpt-6-sol does not trust a lookup result pasted into the caller's message. The first harness run put each lookup result into the caller's turn as a note; VIP read it as the caller's own claim, answered "I don't have a verified order lookup", and missed the facts in all 9 factual replies. No cue tuning fixes that, and loosening "answer only from the tools" to make it pass would be the wrong fix. Evaluate tool-grounded behavior with real tool-call items, a function_call plus its function_call_output, the way the agent's tools answer; the gateway accepts them. (tools/expressive_llm_check.py, conversation())
  • Left alone, the models rarely laugh, and a soft rule is not enough for the small one. In the live run sound cues (laughs, gasps, sighs, breaths) were 1 to 12% of the cues; the models prefer moods and pauses. "Work in at least one sound" fixed Premium and VIP but not Standard (gpt-5-nano), which also wrote [gasp] for [gasps], a cue the filter drops. A firm "Every story MUST include sounds: one at the scare, one on the last line", with the exact cues named, fixed it. The price is formulaic stories.
  • A model asked to laugh may write "ha ha", and the voice reads it aloud. The rule says laughs and noises only as cues from the "Breath and sounds" list, never as words or *laughs*, and the live check counts laughter written out as words (none seen). The rule lives in the cue section, so it reaches only the ElevenLabs Halloween voice.
  • gpt-5-nano invents near-miss cues and ends lines with one unless told not to. Standard wrote calm (four times), gently, closing and a garbled [dramat ic pause], and twice ended a line with [pause], which then stacked with the next line's cue. The filter drops anything off the list (G13), so nothing is read aloud, but the reply loses the cue, and the tool counts the garbled one as a stage direction. Tell the model to use only listed cues and never to end a sentence or a line with one, then check it with tools/expressive_llm_check.py.