VQA Generator

[FaSch87]

Hi,

Thank you for your fantastic work and for open-sourcing it!

I’ve recorded some additional routes and wanted to run the VQA generator. I noticed that it processes only the first 30 samples, which I assume is unintentional or a leftover artifact.

See here:

simlingo/dataset_generation/language_labels/drivelm/carla_vqa_generator.py

Line 66 in 743b243

self.data_boxes_paths_all = glob.glob(os.path.join(self.data_directory, 'data/simlingo/*/*/*/*/boxes/*.json.gz'))[:30]

Best,
Fabian

[FaSch87]

Additionally, i adjusted from commentary.json to commentary_subsentence.json, since commentary.json does not exist.

simlingo/dataset_generation/language_labels/commentary/carla_commentary_generator.py

Line 111 in 743b243

template_file = f"data/augmented_templates/commentary.json"

[FaSch87]

During VQA evaluation using the dataset you have provided, i have observed that 99% of the routes are used for training:

simlingo/simlingo_training/dataloader/dataset_base.py

Line 202 in 743b243

split_percentage = 0.99

For me, this results in only 30 samples used for evaluation which seems to be quite few, compared to the amount of samples provided in evalset_vqa.json. Could you let me what split you have used to produce the numbers reported in the paper, or did you use any sample provided in evalset_vqa.json?