Self-Supervised Learning features for Kaldi ASR.
Kaldi recipes that replace MFCCs with self-supervised (SSL) encoder features such as HuBERT, mHuBERT, MMS and AV-HuBERT. Shared scripts extract the features, reduce them with PCA for the GMM stages, and feed them to Kaldi's GMM-HMM and chain (TDNN-F) training. You can use each pretrained model frozen, or adapt it to the task before extraction.
git clone https://github.com/ialmajai/ssl-kaldi.git
cd ssl-kaldi
conda create -n ssl-kaldi python=3.11 -y
conda activate ssl-kaldi
pip install -r requirements.txt
cd egs/mini_librispeech/s5
./run.sh --stop-stage 12 # needs a Kaldi build; see Getting Started
This downloads Mini LibriSpeech, extracts MMS-300m layer 14 features and trains the GMM ladder. The recipe README covers the chain stage.
The feature extractor is also published on PyPI as
ssl-kaldi-feats. It writes
Kaldi ark/scp features from any Hugging Face speech encoder without a
Kaldi install:
pip install ssl-kaldi-feats
ssl-kaldi-feats --model facebook/hubert-base-ls960 --layer 9 \
--wav-scp data/train/wav.scp --output-dir feats/train
See the package README for multi-layer extraction and library use.
Lip-region tracking from the GRID lipreading recipe (AV-HuBERT visual features).
- Frozen or adapted encoders. The extractor loads a pretrained or a fine-tuned SSL checkpoint the same way. A frozen model needs no training; a LoRA or full fine-tune checkpoint comes from your own training run.
- Straight into Kaldi. Features are written as standard
ark/scp, so Kaldi's hybrid and chain (TDNN-F) pipelines work unchanged. - One toolkit, audio and video. Shared extraction, PCA, and upsampling blocks drive HuBERT, mHuBERT, AfriHuBERT, and AV-HuBERT alike.
- Small datasets. Recipes span English, Arabic, Swahili and visual speech; mini_librispeech trains on 5 hours of audio.
Audio / Video
↓
Pretrained SSL model (PyTorch)
↓
Fine-tuning with a CTC head, LoRA or full (optional)
↓
Frame-level feature extraction
↓
PCA dimensionality reduction / Upsampling (optional)
↓
Kaldi ark/scp features
↓
Standard Kaldi training & decoding
Fine-tuning is optional. egs/grid/s5/run_ctc_finetune.sh fine-tunes AV-HuBERT with LoRA or in full under a CTC head, then exports a plain checkpoint that the extractor reads like a pretrained one.
The shared building blocks live in shared/ (SSL extraction, PCA, interpolation, upsampling) and are reused across every recipe in egs/.
| Recipe | Task | SSL model | Headline result |
|---|---|---|---|
| mini_librispeech | English ASR | MMS / HuBERT | 3.88% WER (dev-clean, chain TDNN, MMS-300m) |
| iqraeval | Arabic phone recognition | mHuBERT | 11.27% PER (TDNN-F) |
| grid | Visual speech recognition (lipreading) | AV-HuBERT | 5.85% WER (unseen speakers, TDNN-F chain) |
| swahili | Swahili ASR | AfriHuBERT / mHuBERT | 19.47% WER (E2E TDNN-F, mHuBERT-147) |
| timit | English phone recognition | MMS / HuBERT | 10.19% PER (core test, chain TDNN) |
A working Kaldi installation is required. Outside Docker, point the recipes at it with export KALDI_ROOT=/path/to/kaldi. Either follow the official Kaldi setup or use the provided Docker images:
# CUDA + Kaldi + SSL feature-extraction dependencies
docker build -t ssl-kaldi docker/
docker run --gpus all -it ssl-kaldi
The docker/ directory also ships Dockerfile.grid, which adds the extra dependencies (dlib, AV-HuBERT) needed for the GRID lipreading recipe.
The Quick Start creates the conda environment. For pinned, reproducible versions install
requirements.lock instead of requirements.txt.
Some recipes have additional requirements; see the per-recipe README. In particular, the GRID lipreading recipe also needs AV-HuBERT, fairseq, and dlib (see its own requirements.txt), and Python 3.8 to 3.10.
Suggestions for improvements or new features are always welcome. Feel free to open an issue or submit a pull request. See CONTRIBUTING.md for development setup, code style, and how to add a recipe.
@misc{ssl_kaldi,
author = {Ibrahim Almajai},
title = {ssl-kaldi: self-supervised speech features for Kaldi ASR recipes},
year = {2025},
howpublished = {\url{https://github.com/ialmajai/ssl-kaldi}},
note = {Accessed: 2026-08}
}
Released under the Apache 2.0 license.
Ibrahim Almajai (ialmajai@gmail.com)