Currently, the project does not support running Whisper with an OpenVINO backend. Adding OpenVINO support could significantly improve inference performance on systems that do not have access to Apple Metal (macOS) or NVIDIA CUDA GPUs.
OpenVINO is Intel’s deep learning optimization and inference toolkit designed to accelerate AI workloads across a wide range of hardware, including Intel CPUs, integrated GPUs, and other accelerators. It provides optimized model execution through graph optimizations, hardware-specific kernels, and an efficient inference runtime.
Supporting OpenVINO would allow the project to leverage these optimizations and potentially provide faster transcription speeds on Intel-based systems and other compatible hardware.
Implementation considerations
- OpenVINO requires different model formats than the default Whisper models.
- Because of this, the implementation would need a runtime switch or configuration option to load the appropriate model depending on whether the OpenVINO backend is enabled.
- The OpenVINO-compatible Whisper models would need to be handled separately from the standard Whisper models.
Build considerations
- The whisper.cpp release page currently does not provide prebuilt OpenVINO binaries for Windows.
- Because of this, users would likely need to build both whisper.cpp and OpenVINO manually to enable this backend.
- Documentation or helper scripts may be needed to simplify the build process, especially on Windows.
Possible scope
- Add an OpenVINO backend option
- Implement model switching between standard Whisper models and OpenVINO-compatible models
- Provide build instructions, especially for Windows
- Optionally add runtime detection for supported hardware
Motivation
This could improve Whisper performance for users running on:
- Intel CPUs
- Intel integrated GPUs
- Systems without CUDA or Apple Metal support
It would broaden hardware compatibility and improve performance for users who currently rely on CPU-only inference.
Currently, the project does not support running Whisper with an OpenVINO backend. Adding OpenVINO support could significantly improve inference performance on systems that do not have access to Apple Metal (macOS) or NVIDIA CUDA GPUs.
OpenVINO is Intel’s deep learning optimization and inference toolkit designed to accelerate AI workloads across a wide range of hardware, including Intel CPUs, integrated GPUs, and other accelerators. It provides optimized model execution through graph optimizations, hardware-specific kernels, and an efficient inference runtime.
Supporting OpenVINO would allow the project to leverage these optimizations and potentially provide faster transcription speeds on Intel-based systems and other compatible hardware.
Implementation considerations
Build considerations
Possible scope
Motivation
This could improve Whisper performance for users running on:
It would broaden hardware compatibility and improve performance for users who currently rely on CPU-only inference.