Skip to content

fix(vad): evaluate the VAD thresholds per analysis frame and honor speech_activity_period - #5486

Open
Kayvan-Zahiri wants to merge 1 commit into
pipecat-ai:mainfrom
Kayvan-Zahiri:fix-vad-per-frame-timing
Open

Kayvan-Zahiri wants to merge 1 commit into
pipecat-ai:mainfrom
Kayvan-Zahiri:fix-vad-per-frame-timing

Conversation

@Kayvan-Zahiri

@Kayvan-Zahiri Kayvan-Zahiri commented Aug 28, 2026 •

Copy link
Copy Markdown

Summary

  • VADAnalyzer checked start_secs and stop_secs once per call rather than once per analysis frame. _run_analyzer drains the whole buffer in a while loop, but both threshold checks sat after it, so a buffer holding several analysis frames could cross a threshold and fall back out of it before either check ran. A confidence dip mid-utterance then wiped the accumulated start count and speech that had already qualified stayed STARTING, with no on_speech_started for the turn. The checks now run inside the loop, which makes one iteration behave exactly like one frame delivered on its own. Chunk size no longer changes the state the analyzer settles on.

Bundled transports use 10 to 20 ms chunks and are unaffected. The WebSocket transports take whatever chunk size the client sends, and the while loop was added in #2184 because create_stream_resampler can emit larger buffers than expected.

Testing

uv run pytest tests/test_vad_analyzer.py tests/test_vad_controller.py tests/test_vad_processor.py

Reverting the source change fails the new analyzer tests at 4 and 12 frames per chunk with VADState.STARTING != VADState.SPEAKING.

I also swept the neighboring inputs. Across 9,660 comparisons of the same confidence script delivered at 2 to 24 analysis frames per chunk, over seven sample rate and threshold combinations including 8 kHz and a 10 ms analysis window, the settled state disagreed with frame by frame delivery 771 times before and 0 times after. Sub-frame and single-frame delivery, the path every bundled transport takes, is unchanged, as is the real Silero model at 16 kHz and 8 kHz, start_secs=0, thresholds that are never reached, the volume gate, empty buffers, and set_params mid-stream. Full suite failure set is identical to main on this machine.

One thing this does not fix: a buffer carrying an entire turn still returns only the state it ends on, so a caller reading one state per call can still miss transitions that begin and end inside one chunk. That is the shape of analyze_audio rather than the threshold placement, and it seemed like a separate conversation.

(This originally also wired speech_activity_period into _handle_audio; #5805 merged that same change on 9/16, so it's been dropped here.)

The analysis frame is the unit the state machine advances on, so the
state the analyzer settles on should not depend on how much audio the
caller hands over at once. `_run_analyzer` drains every analysis frame
in the buffer but checked the start and stop thresholds once per call,
so a buffer holding several frames could cross a threshold and fall back
out of it before either check ran. A dip in confidence mid-utterance
then wiped the accumulated start count, and speech that had already
qualified stayed STARTING, with no `on_speech_started` for the turn. The
WebSocket transports, which take their chunk size from the client, are
the most exposed.
@Kayvan-Zahiri
Kayvan-Zahiri force-pushed the fix-vad-per-frame-timing branch from 73e548e to 9158230 Compare September 21, 2026 21:50
@Kayvan-Zahiri

Copy link
Copy Markdown
Author

Rebased onto main to clear the conflict. #5805 landed the same _maybe_speech_activity() change as my second commit, so I dropped that commit and its changelog entry; this PR is now just the per-frame threshold fix. On current main, 5 of the 7 new tests in tests/test_vad_analyzer.py fail. On this branch the analyzer, controller and processor VAD tests all pass (18).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants