Skip to content

feat(openvino): enable speaker diarization and translation - #547

Open
avanish-garg wants to merge 1 commit into
collabora:mainfrom
avanish-garg:feature/openvino-tensorrt-diarization-translation
Open

avanish-garg wants to merge 1 commit into
collabora:mainfrom
avanish-garg:feature/openvino-tensorrt-diarization-translation

Conversation

@avanish-garg

Copy link
Copy Markdown
Contributor

Fixes #546

What changes are proposed in this pull request?

Diarization and real-time translation are implemented as backend-agnostic
logic in ServeClientBase — update_segments() already calls
self._identify_speaker() (pure audio-slice + pyannote embeddings, no
coupling to any Whisper backend) and pushes completed segments to
self.translation_queue (consumed by a separate M2M100 translation thread,
again independent of the transcription backend). Both already work for the
faster_whisper backend.

ServeClientOpenVINO.handle_transcription_output() already routes through
this same shared update_segments() — so wiring these two constructor
params through was the only gap.

TensorRT is deliberately NOT included. ServeClientTensorRT overrides
speech_to_text()/handle_transcription_output() entirely and never calls
update_segments() at all — its segments are bare {"text": ...} dicts with
no timing info. Enabling diarization/translation there would need a real
restructuring of its transcription loop, not a parameter forward. This is a
genuine architectural difference, verified by reading trt_backend.py
directly, not an oversight.

How is this PR tested?

Added tests/test_openvino_backend.py, including an end-to-end proof test
(test_diarization_and_translation_actually_fire_via_update_segments) that
constructs a real ServeClientOpenVINO with a mock diarizer and a real
queue.Queue, feeds it OpenVINO-shaped segments (start_ts/end_ts per
WhisperDecodedResultChunk), and asserts both the speaker label and the
queued translation segment come out correctly — with zero code changes
beyond the constructor forward.

Full suite after the change: 337 tests (332 existing + 5 new), all passing,
14 skipped (pre-existing, hardware-gated). flake8 --select=E9,F63,F7,F82
(blocking CI check): 0 errors.

Notes for reviewers

No CONTRIBUTING.md/CLA/DCO exists in this repo, so there's no process gate to
satisfy here.

Diarization and real-time translation are implemented as
backend-agnostic logic in ServeClientBase - update_segments() already
calls self._identify_speaker() (pure audio-slice + pyannote embeddings,
no coupling to any Whisper backend) and pushes completed segments to
self.translation_queue (consumed by a separate M2M100 translation
thread, again independent of the transcription backend). Both already
work for the faster_whisper backend.

ServeClientOpenVINO's handle_transcription_output() already routes
through this same shared update_segments() - so wiring these two
constructor params through was the only gap. Confirmed real by test:
TestServeClientOpenVINODiarizationTranslation.
test_diarization_and_translation_actually_fire_via_update_segments
constructs a real ServeClientOpenVINO with a mock diarizer and a real
queue, feeds it OpenVINO-shaped segments, and asserts both the speaker
label and the queued segment come out the other end with zero
additional wiring beyond the constructor forward.

TensorRT is deliberately NOT included: ServeClientTensorRT overrides
speech_to_text() and handle_transcription_output() entirely and never
calls update_segments() at all - its segments are bare {"text": ...}
dicts with no timing info, so diarization/translation would need a
real restructuring of its transcription loop, not a parameter forward.
This is a genuine architectural difference, not an oversight.
@avanish-garg
avanish-garg force-pushed the feature/openvino-tensorrt-diarization-translation branch from d41300e to 0de36a0 Compare October 21, 2026 13:34

@boxerab boxerab left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, the wiring is right: openvino's handle_transcription_output already goes through update_segments, and get_segment_start/get_segment_end read start_ts/end_ts.

one problem: self._create_diarizer(options) is now called inside the openvino try. with known_speakers in the handshake it enrolls them immediately, which loads the pyannote model and can raise (pyannote missing, no hf token for the gated model, download failure). the openvino except Exception catches that, tells the client openvino isn't supported, and sets self.backend = BackendType.FASTER_WHISPER, which is server-wide, so every later client runs on faster_whisper too.

could you create the diarizer once before the backend branches and pass it to both the openvino and faster_whisper constructors? that also stops it being built twice when openvino falls back.

smaller:

  • test_defaults_have_no_diarization_or_translation, test_diarization_and_translation_queue_are_stored and the signature-order test only check attribute storage or parameter names, please drop them. the update_segments test and the server forwarding tests are the useful ones.
  • test_translation_queue_is_forwarded_when_enabled patches threading.Thread globally, patching it in whisper_live.server is enough.
  • a line in the readme diarization section saying it works on faster_whisper and openvino but not tensorrt would help.

@boxerab

boxerab commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

ignore my note about patching threading.Thread, server.py imports the threading module so it's the same patch either way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Enable speaker diarization and translation for the OpenVINO backend

2 participants