CODE issues

[2640754856]

I'm trying to reproduce an experiment by following the code, but the resulting metric WER is very poor, and even the loss function doesn't converge. What's the issue?

[Bindkushal]

I investigated the same reproduction problem. After reading through run.sh and the finetune/inference code, here are the specific causes:

1. LLaMA-2 assumed throughout
   The repo is hardcoded to Llama-2-7b-hf. If you try LLaMA-3 or any other variant, the tokenizer format is different and it fails silently no error, just garbage outputs and WER never improves.

2. downsampling-factor=4 is hardcoded with no guidance
   This factor controls how Whisper decoder states are projected to LLaMA. It's hardcoded to 4 in both finetune.py and inference.py with no explanation of how to tune it. On different GPU memory configurations or batch sizes, this causes the projection to misalign and loss stops converging.

3. No pretrained checkpoint provided
   The repo only gives training code. Reproducing paper results requires training 35 epochs from scratch, which needs significant GPU time and compute. Most users cannot reproduce without the actual checkpoint used in the paper.

4. IndicVoices and non-standard datasets silently produce bad data
   dump_dataset_v2.py only handles FLEURS, CommonVoice, and LibriSpeech. Any other dataset produces malformed .pt files with no error thrown leading to poor WER that looks like a model problem but is actually a data problem.

Suggested fixes:

Add a note in README specifying LLaMA-2 only, with version pinning
Document how to tune downsampling-factor for different hardware
Release the pretrained checkpoint used in the paper
Add a dataset validation check in dump_dataset_v2.py that raises a clear error for unsupported datasets

Happy to open PRs for any of these if maintainers are interested.