Skip to content

feat(speech): split offline STT transcription at VAD gaps - #1607

Merged
msluszniak merged 2 commits into
mainfrom
@ms/stt-vad-chunking
Nov 4, 2026
Merged

msluszniak merged 2 commits into
mainfrom
@ms/stt-vad-chunking

Conversation

@msluszniak

@msluszniak msluszniak commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Description

  • Whisper and Qwen3-ASR transcribe now cut long audio at the last FSMN VAD gap within the model's limit, fall back to the quietest 100 ms for unbroken speech, and skip silence.
  • Shared splitter in speech/utils/audioSplitting.ts.
  • Qwen3-ASR sometimes stops a long piece before writing anything; such a piece is transcribed again in two halves.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  • yarn test (new splitter, Whisper and Qwen3-ASR cases).
  • apps/speech: transcribe a multi-minute recording.

Screenshots

Related issues

Closes #1590

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

WER, host Python port of the pipelines with the shipped XNNPACK .ptes (not on device). Long = 24 LibriSpeech test-clean chapters joined to 77-246 s; short = first 100 clips.

set model old new
long whisper tiny.en 8.00% 5.31%
long whisper base.en 6.52% 3.86%
long Qwen3-ASR 0.6B 2.87% 2.15%
short whisper tiny.en 3.79% 3.75%
short whisper base.en 3.24% 3.41%
short Qwen3-ASR 0.6B 1.88% 2.05%

Short-clip changes are 1-2 word spelling variants, no clipped speech.

Galaxy S26 Ultra (release), tiny.en and Qwen3-ASR on 3 of the long files: within 0-8 words of the host run; 10 s of silence returns "" in ~25 ms.

Whisper and Qwen3-ASR transcribe now cut long audio at the last VAD gap
within the model's limit, fall back to the quietest 100 ms for unbroken
speech, and skip silence.

Closes #1590
@msluszniak msluszniak self-assigned this Nov 4, 2026
@msluszniak msluszniak added the feature PRs that implement a new feature label Nov 4, 2026
@msluszniak
msluszniak marked this pull request as draft November 4, 2026 12:04
Qwen3-ASR now and then stops a long piece before writing anything; when
that happens to a piece of 5 s or more, transcribe its two halves, cut at
the speech gap nearest the middle.
@msluszniak
msluszniak marked this pull request as ready for review November 4, 2026 14:06
@msluszniak
msluszniak requested a review from barhanc November 4, 2026 14:06
@msluszniak
msluszniak requested a review from barhanc November 4, 2026 17:41

@barhanc barhanc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the 2 STT models and the work well.

@msluszniak
msluszniak merged commit d35ad55 into main Nov 4, 2026
7 checks passed
@msluszniak
msluszniak deleted the @ms/stt-vad-chunking branch November 4, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature PRs that implement a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Split offline STT transcription at VAD gaps

2 participants