Repository navigation
fix(vad): evaluate the VAD thresholds per analysis frame and honor speech_activity_period - #5486
Open
Kayvan-Zahiri wants to merge 1 commit into
Open
Kayvan-Zahiri wants to merge 1 commit into
Kayvan-Zahiri wants to merge 1 commit into
Conversation
pradneshfernandez
approved these changes
Sep 11, 2026
The analysis frame is the unit the state machine advances on, so the state the analyzer settles on should not depend on how much audio the caller hands over at once. `_run_analyzer` drains every analysis frame in the buffer but checked the start and stop thresholds once per call, so a buffer holding several frames could cross a threshold and fall back out of it before either check ran. A dip in confidence mid-utterance then wiped the accumulated start count, and speech that had already qualified stayed STARTING, with no `on_speech_started` for the turn. The WebSocket transports, which take their chunk size from the client, are the most exposed.
Kayvan-Zahiri
force-pushed
the
fix-vad-per-frame-timing
branch
from
September 21, 2026 21:50
73e548e to
9158230
Compare
Author
|
Rebased onto main to clear the conflict. #5805 landed the same |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VADAnalyzercheckedstart_secsandstop_secsonce per call rather than once per analysis frame._run_analyzerdrains the whole buffer in awhileloop, but both threshold checks sat after it, so a buffer holding several analysis frames could cross a threshold and fall back out of it before either check ran. A confidence dip mid-utterance then wiped the accumulated start count and speech that had already qualified stayedSTARTING, with noon_speech_startedfor the turn. The checks now run inside the loop, which makes one iteration behave exactly like one frame delivered on its own. Chunk size no longer changes the state the analyzer settles on.Bundled transports use 10 to 20 ms chunks and are unaffected. The WebSocket transports take whatever chunk size the client sends, and the
whileloop was added in #2184 becausecreate_stream_resamplercan emit larger buffers than expected.Testing
uv run pytest tests/test_vad_analyzer.py tests/test_vad_controller.py tests/test_vad_processor.pyReverting the source change fails the new analyzer tests at 4 and 12 frames per chunk with
VADState.STARTING != VADState.SPEAKING.I also swept the neighboring inputs. Across 9,660 comparisons of the same confidence script delivered at 2 to 24 analysis frames per chunk, over seven sample rate and threshold combinations including 8 kHz and a 10 ms analysis window, the settled state disagreed with frame by frame delivery 771 times before and 0 times after. Sub-frame and single-frame delivery, the path every bundled transport takes, is unchanged, as is the real Silero model at 16 kHz and 8 kHz,
start_secs=0, thresholds that are never reached, the volume gate, empty buffers, andset_paramsmid-stream. Full suite failure set is identical tomainon this machine.One thing this does not fix: a buffer carrying an entire turn still returns only the state it ends on, so a caller reading one state per call can still miss transitions that begin and end inside one chunk. That is the shape of
analyze_audiorather than the threshold placement, and it seemed like a separate conversation.(This originally also wired
speech_activity_periodinto_handle_audio; #5805 merged that same change on 9/16, so it's been dropped here.)