feat(voxtral): Voxtral Realtime STT provider with concurrent streaming - #21
Open
tibroc wants to merge 12 commits into
Open
feat(voxtral): Voxtral Realtime STT provider with concurrent streaming#21tibroc wants to merge 12 commits into
tibroc wants to merge 12 commits into
Conversation
tibroc
changed the title
prlanzarin
self-requested a review
June 9, 2026 15:35
Adds `STT_PROVIDER=voxtral-realtime` backed by the vLLM Voxtral Realtime WebSocket API (`/v1/realtime`). - _writer — RMS VAD detects speech; sends opening commit at utterance start, streams 50 ms PCM16 chunks in real time, sends closing commit + commit(final: True) on silence. - _reader — concurrent asyncio task on the same WebSocket; reads transcription.delta → INTERIM, transcription.done → FINAL, loops across utterances without reconnecting. - utterance_start snapshot — captured at first delta so all events share the same BBB transcriptId; prevents later delta bursts from overwriting the visible caption.
The fixed RMS energy threshold cannot distinguish speech from background noise and clips quietly-spoken words. Replace it with Silero VAD, which uses a neural model and is already bundled in livekit-agents[silero]. Python 3.11 is required because onnxruntime ≥ 1.24 dropped Python 3.10 wheels.
Words spoken across the forced 8 s segment split are lost because the cut lands mid-word and neither segment has enough audio context to transcribe it correctly. Two changes address this: - Raise the default max-buffer cap from 8 s to 30 s so forced splits are rare in practice; natural pauses detected by the VAD close segments cleanly before the cap is reached. - Keep a rolling pre-roll across segment boundaries. Previously the pre-roll was cleared on close, so the next segment started cold. Now the last ~0.5 s is replayed as the lead-in of the following segment, giving the model enough context to recover boundary words. The pre-roll length is clamped to min_silence_duration to prevent a normal utterance onset from replaying the previous utterance's tail.
tibroc
marked this pull request as ready for review
June 11, 2026 10:08
Author
|
I had to upgrade python to at least version 3.11 for silero. but i see no problems in that (maybe even update to 3.14?). |
Author
|
Much of this implementation could eventually be extracted into a proper LiveKit plugin (similar to the existing Mistral plugin, which targets the official Mistral API rather than vLLM and is therefore not compatible here). A first-party plugin would let any LiveKit agent use Voxtral Realtime with a single import, rather than carrying the WebSocket protocol logic in the application layer. |
…down Three paths silently lose utterances, all presenting as captions that get an interim (or nothing) and never a final: 1. When the server sends an `error` event or the receive loop fails, the reader task exits while the WebSocket stays alive. The writer keeps appending audio into a session nobody reads, so every subsequent utterance is lost with no log and no recovery until the track cycles. 2. When a speaker mutes or unpublishes right after talking, frames stop before Silero accumulates the 0.6 s of silence needed to fire END_OF_SPEECH. Task cancellation then abandons the open segment: no closing commit is sent (abrupt drops are a known vLLM realtime crash trigger, vllm#34532), the accumulated delta text is discarded, and BBB shows the interim caption as pending forever. Reconnects mid-utterance lose the segment the same way. 3. Even on a clean stream end, teardown cancels the reader immediately after the writer flushes, so the tail segment's transcription.done is never read and its FINAL is dropped. Fix each at the point where the information still exists: - The reader closes the socket when it exits for any reason other than cancellation. The writer's next send then fails and the existing reconnect/backoff path takes over, turning silent death into recovery. The pipeline also catches ConnectionResetError, since a send on a locally-closed socket can surface as that instead of ClientError. - The writer catches CancelledError and best-effort sends the closing commit (1 s cap) for an open segment before propagating. - The in-flight segment's delta text is hoisted to _vad_loop scope so teardown can emit a FINAL from the best available text whenever the real transcription.done will never arrive — converting "caption lost" into "final from partial data" on every loss path at once. - After a clean writer exit, a bounded drain (3 s) lets the reader consume the tail segment's transcription.done before cancellation. An alternative for (1) — a cross-task signal so reconnection does not wait for the writer's next send — was discarded as added coupling for a rare event; the cost is only a degraded onset for the utterance that triggers the reconnect. The duplicated SpeechEvent construction collapses into a single _emit_transcript helper, and the reconnect backoff literals become module constants so the new reconnect regression test does not need a real 1 s sleep. Regression tests cover all three paths via a scripted WebSocket double whose sends fail once closed, mirroring aiohttp.
…e/segment desync The reader pairs each segment's transcription.done with a queued start time in FIFO order. This is only sound if one open->close cycle produces exactly one done; the close sequence sends a bare commit before commit(final), which the vLLM reference client does not, so an extra server-side done per close would silently shift the pairing and mis-stamp every later segment's BBB transcriptId. Probe test 5 (scripts/probe_protocol.py, run against the production server with real speech) settles this empirically: both close shapes yield exactly one done per segment with identical text, but the bare commit delays the done by ~0.35 s because the server processes an extra commit boundary first. Drop it: same behavior, one third of a second less final-caption latency after every pause. Pairing desync remains a silent failure mode if server behavior ever changes, so make it observable: a transcription event arriving with an empty segment_starts queue now logs a warning instead of quietly using a wall-clock fallback timestamp, and teardown logs segments that never received their done.
…er split
A max-buffer split closes a streaming request mid-speech and reopens on
the next frame. The reopened request starts mid-utterance with no
context, and the replayed lead-in is the same rolling pre-roll used for
fresh onsets: capped at min(pre-roll, min-silence) = 0.5 s so a normal
onset cannot replay the previous utterance's tail across a silence gap.
That cap conflates two different situations. The Voxtral Realtime paper
recommends ~1.28 s of left-padding at stream start ("similar to
attention sinks"), so 0.5 s of context is why words right after a split
keep coming out wrong or missing.
Size the rolling buffer for a split overlap (VOXTRAL_SPLIT_OVERLAP_S,
default 1.5 s) and choose the replay length at open time: a reopen after
a cap-triggered close replays the full overlap — mid-speech the buffer
holds only the current utterance, so the longer replay is safe — while
a fresh onset keeps the min-silence-capped pre-roll. An END_OF_SPEECH
observed while no request is open resets the split flag, so an utterance
that ends in the one-frame window after a cap close does not leak the
long replay into the next onset. Segment starts are backdated by the
replayed length, keeping transcriptId timestamps consistent.
The overlap is transcribed twice; duplicated words at split boundaries
are the accepted trade-off for not losing them. The alternative — no
overlap plus client-side stitching of cut words — cannot work, as the
model has no phonetic context to transcribe a word fragment on either
side of the boundary (see notes/progressive-transcription-investigation.md).
Fix, in passing, the buffer trim form: `del preroll[:-preroll_max]`
deletes nothing when the cap is zero, growing the buffer without bound.
Regression tests verified to fail against the previous implementation:
split reopens must replay the byte-exact tail of the prior segment, and
fresh onsets must stay onset-capped even with a full overlap buffer.
…restarts A locale change restarts transcription via stop→start: stop_transcription_for_user cancels the pipeline task and start_transcription_for_user synchronously registers the replacement under the same identity. task.cancel() only schedules the cancellation, so the old task's finally block runs after the replacement is registered — and its unconditional processing_info.pop() removes the replacement's entry. The new pipeline keeps running but is untracked: it can no longer be stopped, and a later start for the same participant spawns a second concurrent pipeline on the same track, producing duplicate transcripts. Guard the pop: deregister only when the stored task is asyncio.current_task(), i.e. when this pipeline still owns the entry. A pipeline that was replaced leaves the replacement's registration alone; a pipeline that ends normally still cleans up after itself. The same stop→start-over-unconditional-pop pattern exists in the OpenAI provider and the base class, but those are maintained separately and are deliberately left untouched on this branch; the equivalent guard should be applied there upstream. The regression test drives a real pipeline through a locale change and was verified to fail against the unguarded implementation.
…the speech band Audio arrives from LiveKit at 48 kHz and the model requires 16 kHz. The conversion uses np.interp — linear interpolation with no low-pass filter — so all energy above the 16 kHz Nyquist (8 kHz) folds back into the speech band as aliasing distortion on every frame. That is a constant, diffuse transcription-accuracy penalty that no amount of protocol tuning can recover. Replace it with rtc.AudioResampler (SoX), which band-limits before decimating. The resampler is streaming — its filter state must persist across frames, as resampling each 10 ms frame independently would reintroduce boundary artifacts — so the per-frame pure function becomes a per-stream _AudioNormalizer instance, drained via flush() at end of stream before the final commit. Mono downmix and the 16 kHz passthrough short-circuit are unchanged. The new anti-aliasing regression test feeds a 10 kHz tone at 48 kHz through the normalizer and requires >20 dB attenuation; the previous implementation fails it, passing the tone through at nearly full energy aliased to 6 kHz.
… startup Two startup/connection hardening changes plus configuration and documentation cleanup: A handshake timeout waiting for session.created is not a reconnectable error: it falls into the generic exception handler and permanently ends transcription for the participant. vLLM spends 2-5 minutes on CUDA-graph warmup after startup, during which the WebSocket may connect but respond slowly — a participant joining in that window loses captions for the whole meeting. Treat TimeoutError like a connection error and retry with the existing backoff. The WebSocket URL falls back to https://api.openai.com/v1 when VOXTRAL_BASE_URL is unset, but OpenAI does not host Voxtral — the fallback can only produce confusing auth/protocol failures at runtime. Require base_url at agent creation and fail with an actionable message. Remove the VOXTRAL_TARGET_SAMPLE_RATE variable: the model requires 16 kHz input unconditionally, so configurability is purely a footgun. The model card mandates temperature 0.0, but vLLM's session.update shape for temperature is undocumented; sending it blind risks an error event per connection. Add probe test 6 to determine empirically which shape (if any) the server accepts before wiring it into the provider. Update README: Python 3.11 requirement (raised when Silero VAD was introduced), Voxtral Realtime in the supported engines, and a provider configuration section. Fix the module docstring's stale claim that a commit is needed to trigger generation.
…date
The Voxtral model card mandates "always set the temperature to 0.0" —
greedy decoding is required for deterministic, stable transcription —
but session.update only carries the model, leaving sampling temperature
to whatever the server defaults to.
vLLM's session.update shape for temperature is undocumented, so the
field was probed rather than sent blind (probe test 6): the server
accepts temperature at the TOP level alongside model and still
transcribes normally, while the OpenAI-style nested
{"session": {...}} shape is rejected with "Missing required field:
model" — incidentally hard-confirming the flat-shape assumption the
provider was built on. Whether the server honors or ignores the field
is not observable from the protocol; sending it is at worst a no-op.
…arrives
vLLM's realtime handler runs one generation per connection and silently
drops any commit that arrives while the previous segment's generation is
still running ("Generation already in progress, ignoring commit"). The
writer sends commits on VAD/cap timing alone, so two paths collide with
this: a max-buffer split reopens on the very next frame — while the
server is guaranteed to still be decoding the closed segment — and a
back-to-back utterance can open before the previous done (the decode
tail takes 0.5–2 s). A dropped opener costs the segment its live
interim captions (the audio is batch-transcribed at the closing commit
instead); a dropped closer loses the segment's transcription.done
entirely, desyncing the FIFO start-time pairing so later captions merge
or overwrite each other. The damage compounds within a connection's
lifetime, which is why transcription quality degrades over time and
hits long utterances hardest.
Track the number of opened segments whose transcription.done has not
been read yet and defer opening commits while it is non-zero. Audio
arriving during the wait accumulates in a gate buffer (seeded with the
usual onset/split lead-in) and is replayed once the segment opens, so
no audio is lost — captions for the gated segment just start slightly
later. An utterance that ends while gated is closed right after it
finally opens. A bounded timeout guards against a done that never
arrives (an already-desynced session): the counter is resynced and the
open proceeds ungated, i.e. the previous behavior.
An alternative considered was retrying the opener until the server
stops warning, but the server gives the client no feedback when it
ignores a commit — the drop is only visible in server logs — so the
client must serialize commit cycles itself.
Member
|
@tibroc I've merged the stt/refactor/generic-providers branch into |
tibroc
changed the base branch from
stt/refactor/generic-providers
to
development
July 31, 2026 13:04
Author
|
@prlanzarin ready for review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
STT_PROVIDER=voxtral-realtimebacked by the vLLM Voxtral Realtime WebSocket API (/v1/realtime). I developed this against a mistralai/Voxtral-Mini-4B-Realtime-2602 running in vLLM.Findings about the realtime endpoint
The server only streams transcription.delta tokens in real time if the client sends an opening input_audio_buffer.commit before any audio. Without it the server silently buffers everything and batch-transcribes on the closing commit — producing multi-second delays regardless of any client-side optimisation
Architecture
ToDos:
Setup
Running Voxtral Mini in VLLM
I run mistralai/Voxtral-Mini-4B-Realtime-2602 in vLLM using a custom built docker container that contains vllm and the dependencies for running voxtral: https://github.com/virtUOS/vllm-voxtral. (Note: this container can be used with
STT_PROVIDER=voxtral-realtimeas well asSTT_PROVIDER=openai).For the test setup I got good results with a nvidia RTX 3090 Ti (24 GB) with this start command:
For other graphic cards the optimal start parameters probably differ. You can still optimize especially
--max-model-lenand--max-num-batched-tokensfor the smoothest experience.Config for bbb-livekit-stt
You can run this
docker-compose.ymlon your bbb server for testeing (it uses the pre-built container images from my fork):