Repository navigation
Call mode: the bot as a participant in a LiveKit room - #217
Merged
Merged
Conversation
voice_turn() runs one turn for any media backend: deltas to a send function, the whole reply posted to the room, the turn recorded for a report. voice_turn_report() truncates a recorded turn to what was heard. Converse and ReportTurn now call these; their wire behaviour is unchanged. A speaker can be named in front of the words, for a call with more than one person in it.
In a call the agent hears one PCM stream per participant and must decide where an utterance starts and ends itself, since the transcription route takes one file per utterance. call-audio.R has the energy detector (adaptive floor, slow rise on sound, fast fall on quiet, preroll, minimum and maximum length), WAV in and out, the sentence split for synthesis a piece at a time, and the count of what was heard when playback stops. call-media.R posts the OpenAI-shaped transcription and synthesis routes with curl, configured under voice.stt and voice.tts, through an HTTP hook the tests replace.
One participant in a call, over four collaborators the loop does not own (the SFU leg, transcription, synthesis, the voice brain), so each is faked in the tests and real in a deployment. Audio per participant is cut into utterances; one at a time an utterance is transcribed and, when the policy says so, answered: whole sentences are synthesized and published as the brain streams them, in 200 ms chunks with a poll between, so someone talking over the agent is heard within a chunk. That stops publishing, cancels the generation through the brain's send contract, and reports how much was heard so the room record is trimmed to it. With several people the agent answers when addressed by name, or the same person's follow-up within 20 s; with one, always. call-adapters.R binds the loop to livekitr, to voice_turn() over a voice state, and to the speech routes.
The bot process long-polls Matrix and owns the room's crypto store; a call needs audio polled every few tens of milliseconds and blocks on transcription and synthesis. So the call loop runs in a child R process, and what crosses between the two goes through files in the call's directory: the child asks the parent to post a reply or edit a post and waits for the event id; the parent sends the child media keys and the order to leave. The child holds the LiveKit token and the model credentials, never a Matrix credential. Checked against a LiveKit dev server at home: the process joins, hears a person, cuts the utterance, reports the (unreachable) transcription route's failure through the mailbox, and leaves on command.
The configuration vignette gains the voice.stt, voice.tts and voice.call keys; CLAUDE.md the one-brain-two-backends picture and the new files; NEWS the entry. The rgrpc floor test allows the installed Suggests field to wrap after ">=", which adding livekitr made it do.
The bot process owns the Matrix account, so it joins the call's signaling through chat.api's call surface (chat_call_join: membership, the media token, the per-member keys over Olm; chat.api 0.1.0.1, an encrypted client) and starts the call worker from what the join returned: the token, our key first and the peers' after, 30 messages of history, the names the agent answers to, and a config stripped of every Matrix credential. Each poll, bot_call_pump() forwards the keys that arrived as worker key commands, answers the worker's post and edit requests through the room, and ends the call when the worker dies. /clear stops a session's call with the session. While a call is on, bot_run_step() cuts the long poll to half a second so a reply is posted promptly. chat.api Suggests floor to 0.1.0.1. Not yet tried against a real SFU: the fleet has none.
The OpenAI-shaped router in front of the fleet's GPU hosts is not built
yet, and the hosts themselves answer today: a gpu.ctl host takes one
POST /infer with the protocol, the catalog entry, a content-hash key
and the input, where whisper takes audio_b64 and answers an {ok, value}
envelope, and chatterbox takes text and the reference clip and answers
float32 PCM with the rate in the X-GpuHost-Meta header. voice.stt.wire
and voice.tts.wire pick the shape, default "openai"; key_file names
the host's token, read at each request so a rotation needs no restart;
on this wire voice.tts.voice is the path of the reference WAV, read
once. The key is derived as gpu.host derives it, under corteza's own
prefix. Checked live against troy-g5: a sentence synthesized and
transcribed back.
Two knobs under voice.call for debugging a node: livekit_log sets livekitr's log level in the worker, and every message() the worker emits (livekitr reports that way) becomes a note the bot process logs, since nobody reads a child's stderr; ice_transport = "relay" makes a node that only serves media through TURN fail fast with the relay error instead of waiting out host candidates that can never pair. The worker also notes the URL it is connecting to.
A call button posts a call-membership state event and nothing else, so a bot that waits for /call never hears the call it was invited to. chat.api 0.1.0.1 reports from the poll the rooms where someone other than the bot announced or withdrew a membership; on a notice with someone in, in a room voice.call.rooms allows (absent: any room the bot is in), the bot joins as /call would and says so. When the call's own membership says nobody but the bot is left, it hangs up. Not on the baseline sync: a membership that predates the bot is not an invitation. voice.call.auto_join = false keeps /call as the only way in.
One line per room event (who came and went, tracks published and subscribed, encryption state) and, every ten seconds per participant, how much audio arrived and its peak level. The first live call had the agent in the room for three minutes without a word heard, and nothing in the log to say whether a track was never subscribed, arrived silent (a key it did not have, a muted microphone), or was heard and dropped.
An undecryptable track is almost always a key filed under another identity or index. The bot process now logs, on joining, its own key index and the peers' identities and indices it hands the worker; on each poll, every peer key received and own rotation; and the call's membership when re-read. The worker logs each key it applies, with the identity and index and the key's length.
voice.call.greeting is spoken once the worker is in the call, through the same sayer as a reply (sentence at a time, barge-in stops it), posting nothing and recording no turn. It tells the people in the call the agent is there, and it exercises the agent's own track and media key before anyone has spoken: with a caller whose microphone is dead, it is the only way to learn whether the agent can be heard.
voice.call.connect_delay_s holds the worker between the membership and the media connection. FluffyChat 2.10 clears its ring, plays, and shares its key only for participants who connect after it does; an agent that arrives first is never noticed.
CI installs the Matrix stack but not livekitr (a Suggests), and bot_call_unavailable refuses a join when livekitr is absent, so the auto-join and refusal blocks never reached the fake worker and started[[1L]] was out of bounds. The blocks inject a fake start, so the real worker's native dependency is beside the point; stub the presence check in the namespace redirect while keeping the caps and voice-config checks the refusal test exercises. Verified against a library with livekitr removed.
Creating a bot session validates the provider key, and CI has no ANTHROPIC_API_KEY, so the auto-join block errored once it reached session setup. Set a fake key over the block (the turns are all faked, so nothing calls the model) and restore it after, so a later file's key-absent test still sees it unset. Verified with the full suite run against a library without livekitr and with no API key set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bot joins a room's MatrixRTC call (what Element Call and FluffyChat 2.10 start), hears each participant, transcribes utterances, answers through the same brain the room uses, and speaks the reply. Replaces the gpu.ctl
gpu_voice.protomedia front for calls; the 1:1 AgentVoice mode is untouched and shares the brain.One brain, two media backends
R/voice-turn.R:voice_turn()/voice_turn_report()pulled out of the gRPC handlers so a call can use them.send(delta)returns TRUE when delivered; anything else cancels the generation.R/call-audio.R: energy VAD with an adaptive floor, WAV encode/decode, sentence splitting, the heard-offset rule.R/call-media.R: STT (/v1/audio/transcriptions, multipart) and TTS (/v1/audio/speech, wav) over curl, OpenAI-shaped, configured undervoice.stt/voice.tts.R/call-loop.R: hear, transcribe, think, speak. Per-participant VAD; answers when addressed by name (word boundary) or within 20 s of its last turn, or always with one human; speech published in 200 ms chunks with a poll between, where barge-in is noticed; the room post is trimmed to what was heard. Takes media, stt, tts and brain as arguments;R/call-adapters.Rbinds livekitr and the real brain.R/call-worker.R: the loop in a callr child with a file mailbox (child -> parent: post, edit, note; parent -> child: key, leave). The child holds the LiveKit JWT and model credentials, never a Matrix credential.R/call-bot.R:/calljoins through chat.api's call surface (chat_call_join, chat.api 0.1.0.1, an encrypted client), starts the worker from the token and keys, forwards the keys each poll brings, posts and edits for the worker, ends the call when the worker dies;/hangupleaves;/clearstops the session's call. The long poll is cut to 500 ms while a call is on.Dependencies:
livekitradded to Suggests; mx.client floor 0.2.1.1; chat.api floor 0.0.1.24 -> 0.1.0.1 (.CHAT_API_MINtoo). Documented invignette("configuration")("Voice in a call") and CLAUDE.md.Tests: 6 new files, 264 assertions (271 with a LiveKit dev server) with fakes for media, STT, TTS, brain, chat.api and the worker; the worker test also runs against a LiveKit dev server when
LIVEKITR_TEST_URLis set. Full suite 5273/0 locally. Live check: two processes in a Docker livekit-server room, the agent heard, answered, was interrupted mid-reply, trimmed the post and answered the interruption.CI note: this needs cornball-ai/chat.api#25. The workflow installs chat.api from its default branch, so "Verify Matrix stack" stays red until that merges, which is what the step is for.
Not yet tried against a real homeserver and SFU; the fleet has no SFU node, lk-jwt-service, or
rtc_transportsyet.