SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis

Xiangyue Zhang*, Jianfang Li*, Jiaxu Zhang, Ziqiang Dang, Jianqiang Ren, Liefeng Bo, Zhigang Tu†

📣 Updates

• [2025.09.11] 🔥 Release: - [✔] Inference code - [✔] Training code

💖 Inference Data

If you would like to compare your paper’s results with SemTalk but find it too difficult to run the repository, you can simply download the test .npz file from Google Drive.

⚒️ Quick Start

Build Environtment

We Recommend a python version =3.8 and cuda version >=12.1. Then build environment as follows:

# [Optional] Create a virtual env
conda create -n SemTalk python=3.8
conda activate SemTalk
# pytorch, torchvison
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
# Install with pip:
pip install -r requirements.txt
pip install -U whisperx
sudo apt-get update && sudo apt-get install -y ffmpeg  # if you don't have ffmpeg

Download Data

please refer to EMAGE and download datasets from BEAT2 for datasets and place the dataset inside the SemTalk folder, i.e. path-to-SemTalk/BEAT2

Download Weights

Download hubert-large-ls960-ft(used for extracting audio features), faster-whisper-large-v3(used for inference) weights and place itside in the SemTalk folder, i.e. path-to-SemTalk/facebook. If you are in China, you can use hf-mirror for faster and more reliable downloads. The process may take some time, so please be patient.

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --resume-download facebook/hubert-large-ls960-ft --local-dir facebook/hubert-large-ls960-ft

huggingface-cli download --resume-download Systran/faster-whisper-large-v3 --local-dir Systran/faster-whisper-large-v3

Download pretrained models and weights from google drive, unzip and place it in the SemTalk folder, i.e. path-to-SemTalk/weights.

Finally, these SemTalk folder should be orgnized as follows:

.
├── BEAT2
│   └── beat_english_v2.0.0
├── configs
├── dataloaders
├── datasets   (this folder will appear after you generate the datasets)
│   ├── beat2_cache2
│   ├── beat2_semtalk_train
│   └── semtalk_dataloader.py
├── facebook
│   └── hubert-large-ls960-ft
├── models
├── optimizers
├── src
├── Systran
│   └── faster-whisper-large-v3
├── utils
├── weights
│   ├── pretrained_vq
│   ├── smplx_models
│   ├── best_semtalk_base.bin
│   └── best_semtalk_sparse.bin
├── ae_trainer.py
├── aelower_trainer.py
├── aelowerfoot_trainer.py
├── requirements.txt
├── semtalk_base_trainer.py
├── semtalk_sparse_trainer.py
└── train.py

Generate Dataset

Notice: Please make sure you are in the root directory, i.e. path-to-SemTalk.

1. To generate the training dataset, run:

python dataloaders/save_train_dataset.py # generate train dataset

This process may take some time, so please be patient.

2. To generate the test dataset, run:

python dataloaders/save_test_dataset.py # generate test dataset

🚀Training, Testing, and Inference

Training of SemTalk

Train RVQ-VAE

You can either train your own RVQ-VAE weights and place them under path-to-SemTalk/weights using the commands below, or simply use our pretrained weights.

python train.py --train_rvq --config configs/cnn_vqvae_face_30.yaml # face

python train.py --train_rvq --config configs/cnn_vqvae_hands_30.yaml # hands

python train.py --train_rvq --config configs/cnn_vqvae_upper_30.yaml # upper body

python train.py --train_rvq --config configs/cnn_vqvae_lower_foot_30.yaml # lower foot

python train.py --train_rvq --config configs/cnn_vqvae_lower_30.yaml # lower body

Stage1: Base Motion Generation

python train.py --config configs/semtalk_base.yaml

Notice: Once you have obtained the optimal base motion generation weights, please update the path field to base_ckpt in configs/semtalk_sparse.yaml.

Stage2: Sparse Motion Generation

python train.py --config configs/semtalk_sparse.yaml

Testing of SemTalk

Notice: Before running the test code, make sure the load_ckpt and base_ckpt paths in configs/semtalk_sparse.yaml are set correctly.

python train.py --test_state --config configs/semtalk_sparse.yaml

Inference

you can put your inference wav format aduio on ./demo path, for example, you can run:

python train.py --inference --config configs/semtalk_sparse.yaml --audio_infer_path ./demo/2_scott_0_1_1.wav

📺 Visualize

Following EMAGE, you can download SMPLX blender addon, and install it in your blender 3.x or 4.x. Click the button Add Animation to visualize the generated smplx file (like xxx.npz).

🙏 Acknowledgments

Thanks to EMAGE, DiffSHEG, our code is partially borrowing from them. Please check these useful repos.

📖 Citation

If you find our code or paper helps, please consider citing:

@article{zhang2024semtalk,
  title={SemTalk: Holistic Co-speech Motion Generation with Frame-level Semantic Emphasis},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Dang, Ziqiang and Ren, Jianqiang and Bo, Liefeng and Tu, Zhigang},
  journal={arXiv preprint arXiv:2412.16563},
  year={2024}
}