About dataset

[kaiw7]

Hi, could I know if the model is only trained in the image-audio data? Did you use the video-audio data? I didn't find such data in the shared AVQA. Thanks a lot!

[HarryHsing]

Hi, the released model is trained on AVQA-R1-6K, which is an image–audio dataset.
In addition, I’ve also added a video–audio training feature and tested it on the Video-R1 dataset.
The shared AVQA dataset itself only contains image–audio pairs.

[kaiw7]

Hi, could you share the script for video-audio training?

[HarryHsing]

Hi, I didn’t keep the previous video script, but you can easily recreate it by following the existing structure:

• Referring to grpo.py#L276
  — just change the data_type to "video" in your own video training data json.

• Update the prefix at grpo_trainer.py#L330 for your video training data path.

• The rest of the script can follow the same pattern as the image/audio AVQA example script.

That should give you a working video training script with minimal changes.

[kaiw7]

Hi, thank you very much for your kind reply. I noticed that the Video-R1 dataset only has the video and image, without audio. Could I know how did you test your model on the Video-R1 dataset?

[HarryHsing]

Hi, thank you very much for your kind reply. I noticed that the Video-R1 dataset only has the video and image, without audio. Could I know how did you test your model on the Video-R1 dataset?

Hi, yes, I only used the Video-R1 training dataset for training.
In addition, I also tested training on ViDi and AV-TAU, both of which include audio-visual data.

I did not run evaluations on these datasets, but AVATAR provides extensive evaluations of EchoInk-R1 across many audio-visual benchmarks.