Initial roadmap

[unilight]

Training recipes

• BVCC (24/05/14)
• SOMOS
• NISQA
• TMHINT-QI
• SingMOS (VMC'24 track2)
• PSTN
• Tencent

Benchmarks

• VMC'22 OOD track - having difficulty downloading from BC server now...
• VMC'23
• Zoomed-in BVCC (VMC'24 track1)

Classic models

Non-intrusive (single-ended)

• LDNet https://github.com/unilight/LDNet (24/05/14)
• SSL-MOS https://github.com/nii-yamagishilab/mos-finetune-ssl (24/05/22)
• UTMOS https://github.com/sarulab-speech/UTMOS22 (24/06/12)
• RAMP https://arxiv.org/abs/2308.16488

Do I want to implement these methods?

• PAM https://github.com/soham97/PAM
• SpeechLMScore https://github.com/soumimaiti/speechlmscore_tool

Intrusive (double-ended)

• SVSNet https://ieeexplore.ieee.org/document/9716822

Experimental features

Output

• continuous with L1/L2 loss
• discrete (categorial) with cross-entropy loss

Input feature

• semantic SSL
• Linguistic representation from ASR
  • Whisper PPG
  • sxliu PPG
• audio codec
• general audio representation
  • Supported in S3PRL
    • SSAST
  • CLAP https://github.com/microsoft/CLAP
  • Audio-MAE https://github.com/facebookresearch/AudioMAE
  • BEATS https://github.com/microsoft/unilm/tree/master/beats

Improvements

• In training loop, automatically save models with good results on dev set
• Model ensemble
• Model averaging
• Inference with outside pre-trained models
• Upload/download/inference with, models trained in this toolkit (to where? HuggingFace?)