Repository navigation
feat(openvino): enable speaker diarization and translation - #547
avanish-garg wants to merge 1 commit into
Conversation
Diarization and real-time translation are implemented as
backend-agnostic logic in ServeClientBase - update_segments() already
calls self._identify_speaker() (pure audio-slice + pyannote embeddings,
no coupling to any Whisper backend) and pushes completed segments to
self.translation_queue (consumed by a separate M2M100 translation
thread, again independent of the transcription backend). Both already
work for the faster_whisper backend.
ServeClientOpenVINO's handle_transcription_output() already routes
through this same shared update_segments() - so wiring these two
constructor params through was the only gap. Confirmed real by test:
TestServeClientOpenVINODiarizationTranslation.
test_diarization_and_translation_actually_fire_via_update_segments
constructs a real ServeClientOpenVINO with a mock diarizer and a real
queue, feeds it OpenVINO-shaped segments, and asserts both the speaker
label and the queued segment come out the other end with zero
additional wiring beyond the constructor forward.
TensorRT is deliberately NOT included: ServeClientTensorRT overrides
speech_to_text() and handle_transcription_output() entirely and never
calls update_segments() at all - its segments are bare {"text": ...}
dicts with no timing info, so diarization/translation would need a
real restructuring of its transcription loop, not a parameter forward.
This is a genuine architectural difference, not an oversight.
d41300e to
0de36a0
Compare
boxerab
left a comment
There was a problem hiding this comment.
thanks, the wiring is right: openvino's handle_transcription_output already goes through update_segments, and get_segment_start/get_segment_end read start_ts/end_ts.
one problem: self._create_diarizer(options) is now called inside the openvino try. with known_speakers in the handshake it enrolls them immediately, which loads the pyannote model and can raise (pyannote missing, no hf token for the gated model, download failure). the openvino except Exception catches that, tells the client openvino isn't supported, and sets self.backend = BackendType.FASTER_WHISPER, which is server-wide, so every later client runs on faster_whisper too.
could you create the diarizer once before the backend branches and pass it to both the openvino and faster_whisper constructors? that also stops it being built twice when openvino falls back.
smaller:
test_defaults_have_no_diarization_or_translation,test_diarization_and_translation_queue_are_storedand the signature-order test only check attribute storage or parameter names, please drop them. the update_segments test and the server forwarding tests are the useful ones.test_translation_queue_is_forwarded_when_enabledpatchesthreading.Threadglobally, patching it inwhisper_live.serveris enough.- a line in the readme diarization section saying it works on faster_whisper and openvino but not tensorrt would help.
|
ignore my note about patching threading.Thread, server.py imports the threading module so it's the same patch either way. |
Fixes #546
What changes are proposed in this pull request?
Diarization and real-time translation are implemented as backend-agnostic
logic in
ServeClientBase—update_segments()already callsself._identify_speaker()(pure audio-slice + pyannote embeddings, nocoupling to any Whisper backend) and pushes completed segments to
self.translation_queue(consumed by a separate M2M100 translation thread,again independent of the transcription backend). Both already work for the
faster_whisper backend.
ServeClientOpenVINO.handle_transcription_output()already routes throughthis same shared
update_segments()— so wiring these two constructorparams through was the only gap.
TensorRT is deliberately NOT included.
ServeClientTensorRToverridesspeech_to_text()/handle_transcription_output()entirely and never callsupdate_segments()at all — its segments are bare{"text": ...}dicts withno timing info. Enabling diarization/translation there would need a real
restructuring of its transcription loop, not a parameter forward. This is a
genuine architectural difference, verified by reading
trt_backend.pydirectly, not an oversight.
How is this PR tested?
Added
tests/test_openvino_backend.py, including an end-to-end proof test(
test_diarization_and_translation_actually_fire_via_update_segments) thatconstructs a real
ServeClientOpenVINOwith a mock diarizer and a realqueue.Queue, feeds it OpenVINO-shaped segments (start_ts/end_tsperWhisperDecodedResultChunk), and asserts both the speaker label and thequeued translation segment come out correctly — with zero code changes
beyond the constructor forward.
Full suite after the change: 337 tests (332 existing + 5 new), all passing,
14 skipped (pre-existing, hardware-gated).
flake8 --select=E9,F63,F7,F82(blocking CI check): 0 errors.
Notes for reviewers
No CONTRIBUTING.md/CLA/DCO exists in this repo, so there's no process gate to
satisfy here.