Hi, I have a question that why the number of test videos in 'msvd_test_sequencelabel.h5' is 470?

[tiesanguaixia]

I remember the the standard split contains 1.2K training videos, 100 validation videos, and 670 test videos, so why you did not use the other 200 test videos? Thank u!

[ylqi]

There must be some unexpected mistakes in this repo. Please try the new uploaded msvd_test_sequencelabel.h5 and let me know if you still have any questions.

[tiesanguaixia]

There must be some unexpected mistakes in this repo. Please try the new uploaded msvd_test_sequencelabel.h5 and let me know if you still have any questions.

I run the test.sh again and a new error occurs:

2022-10-16 15:09:06,593:INFO: Inference time: 0.001063, batch_size: 32
2022-10-16 15:09:09,364:INFO: ===> Finished loading epoch 0
2022-10-16 15:09:09,414:INFO: Inference time: 0.000896, batch_size: 32
2022-10-16 15:09:11,936:INFO: >>> Language evaluating ...
Traceback (most recent call last):
  File "test.py", line 123, in <module>
    checkpoint_opt = checkpoint['opt']
  File "/home/tianmingkai/GL-RG-TSGX/train.py", line 330, in test
    results = validate(model, criterion, loader, opt)
  File "/home/tianmingkai/GL-RG-TSGX/train.py", line 306, in validate
    lang_stats = utils.language_eval(loader.cocofmt_file, tmp_checkpoint_json)
  File "/home/tianmingkai/GL-RG-TSGX/utils.py", line 271, in language_eval
    cocoRes = coco.loadRes(pred_file)
  File "/home/tianmingkai/GL-RG/coco-caption/pycocotools/coco.py", line 279, in loadRes
    'Results do not correspond to current coco set'
AssertionError: Results do not correspond to current coco set

It seems like the new error is caused by the new msvd_test_sequencelabel.h5 file. Could you please take some time to run the train and test code of this repo and fix the bugs? Thank you a lot!

[ylqi]

Please check the new uploaded files in data/metadata.
I can successfully run the test code when the msvd_test_cocofmt.json and msvd_test_evalscores.pkl were replaced.

[tiesanguaixia]

Please check the new uploaded files in data/metadata. I can successfully run the test code when the msvd_test_cocofmt.json and msvd_test_evalscores.pkl were replaced.

Thank you a lot! It works.
By the way, may I ask you how you get the video features like msvd_test_gl_feats.h5 with the long_range, short_range and local encoder mentioned in your paper? Is there any code about the first feature extracting stage in this repo? Thank u!

[ylqi]

As a plug-and-play module, the structure of the long_range, short_range and local-keyframe encoder is very flexible, so you can quickly reproduce them with any similar networks you want.

For efficiency, we suggest you directly adopt some simple alternative models on MSVD/MSR-VTT for 1700-dim features (1700 scores of 300 words + 400 actions + 1000 categories). Here are some Quick Start guides:

• Long-range encoder: a video multi-label prediction network, e.g., https://www.dropbox.com/sh/xbdtcr8kwj7m7jd/AADZzaX_Ig_pa3RbywpVW6K-a?dl=0. Make the 300-word dataset (e.g., https://www.dropbox.com/sh/3q7l0ooc52neb0a/AABIFuGoC72ExYohVph1Dz7Qa?dl=0) on MSVD/MSR-VTT. Train the above lightweight network on this video-to-words dataset. Predict the probability distribution of those 300 words for each video.
• Short-range encoder: a video action recognition network, e.g., https://github.com/mzolfaghari/ECO-efficient-video-understanding. Download the ECO code and network weights (pre-trained on the Kinetics-400 dataset). Predict the probability distribution of those 400 actions for each video.
• Local-keyframe encoder: an image classification network, e.g., https://github.com/facebookresearch/ResNeXt. Download the ResNeXt code and network weights (pre-trained on the ImageNet-1000 dataset). Predict the probability distribution of those 1000 categories for each video.

You can use any multi-label prediction, action recognition, and visual classification networks to extract those features. We believe this will be easier and barely impact performance.

[tiesanguaixia]

As a plug-and-play module, the structure of the long_range, short_range and local-keyframe encoder is very flexible, so you can quickly reproduce them with any similar networks you want.

For efficiency, we suggest you directly adopt some simple alternative models on MSVD/MSR-VTT for 1700-dim features (1700 scores of 300 words + 400 actions + 1000 categories). Here are some Quick Start guides:

• Long-range encoder: a video multi-label prediction network, e.g., https://www.dropbox.com/sh/xbdtcr8kwj7m7jd/AADZzaX_Ig_pa3RbywpVW6K-a?dl=0. Make the 300-word dataset (e.g., https://www.dropbox.com/sh/3q7l0ooc52neb0a/AABIFuGoC72ExYohVph1Dz7Qa?dl=0) on MSVD/MSR-VTT. Train the above lightweight network on this video-to-words dataset. Predict the probability distribution of those 300 words for each video.
• Short-range encoder: a video action recognition network, e.g., https://github.com/mzolfaghari/ECO-efficient-video-understanding. Download the ECO code and network weights (pre-trained on the Kinetics-400 dataset). Predict the probability distribution of those 400 actions for each video.
• Local-keyframe encoder: an image classification network, e.g., https://github.com/facebookresearch/ResNeXt. Download the ResNeXt code and network weights (pre-trained on the ImageNet-1000 dataset). Predict the probability distribution of those 1000 categories for each video.

You can use any multi-label prediction, action recognition, and visual classification networks to extract those features. We believe this will be easier and barely impact performance.

Thank you for your detailed reply! I really appreciate it. I also want to ask that how many local key frames do you get from a video and how you sample the local key frame(s) from lots of frames in a single video. Sorry to disturb you. Thanks a lot!

[ylqi]

As a plug-and-play module, the structure of the long_range, short_range and local-keyframe encoder is very flexible, so you can quickly reproduce them with any similar networks you want.
For efficiency, we suggest you directly adopt some simple alternative models on MSVD/MSR-VTT for 1700-dim features (1700 scores of 300 words + 400 actions + 1000 categories). Here are some Quick Start guides:

• Long-range encoder: a video multi-label prediction network, e.g., https://www.dropbox.com/sh/xbdtcr8kwj7m7jd/AADZzaX_Ig_pa3RbywpVW6K-a?dl=0. Make the 300-word dataset (e.g., https://www.dropbox.com/sh/3q7l0ooc52neb0a/AABIFuGoC72ExYohVph1Dz7Qa?dl=0) on MSVD/MSR-VTT. Train the above lightweight network on this video-to-words dataset. Predict the probability distribution of those 300 words for each video.
• Short-range encoder: a video action recognition network, e.g., https://github.com/mzolfaghari/ECO-efficient-video-understanding. Download the ECO code and network weights (pre-trained on the Kinetics-400 dataset). Predict the probability distribution of those 400 actions for each video.
• Local-keyframe encoder: an image classification network, e.g., https://github.com/facebookresearch/ResNeXt. Download the ResNeXt code and network weights (pre-trained on the ImageNet-1000 dataset). Predict the probability distribution of those 1000 categories for each video.

You can use any multi-label prediction, action recognition, and visual classification networks to extract those features. We believe this will be easier and barely impact performance.

Thank you for your detailed reply! I really appreciate it. I also want to ask that how many local key frames do you get from a video and how you sample the local key frame(s) from lots of frames in a single video. Sorry to disturb you. Thanks a lot!

For the Local-Keyframe Encoder, we select 30 keyframes evenly in the time sequence for each video.

[tiesanguaixia]

As a plug-and-play module, the structure of the long_range, short_range and local-keyframe encoder is very flexible, so you can quickly reproduce them with any similar networks you want.
For efficiency, we suggest you directly adopt some simple alternative models on MSVD/MSR-VTT for 1700-dim features (1700 scores of 300 words + 400 actions + 1000 categories). Here are some Quick Start guides:

• Long-range encoder: a video multi-label prediction network, e.g., https://www.dropbox.com/sh/xbdtcr8kwj7m7jd/AADZzaX_Ig_pa3RbywpVW6K-a?dl=0. Make the 300-word dataset (e.g., https://www.dropbox.com/sh/3q7l0ooc52neb0a/AABIFuGoC72ExYohVph1Dz7Qa?dl=0) on MSVD/MSR-VTT. Train the above lightweight network on this video-to-words dataset. Predict the probability distribution of those 300 words for each video.
• Short-range encoder: a video action recognition network, e.g., https://github.com/mzolfaghari/ECO-efficient-video-understanding. Download the ECO code and network weights (pre-trained on the Kinetics-400 dataset). Predict the probability distribution of those 400 actions for each video.
• Local-keyframe encoder: an image classification network, e.g., https://github.com/facebookresearch/ResNeXt. Download the ResNeXt code and network weights (pre-trained on the ImageNet-1000 dataset). Predict the probability distribution of those 1000 categories for each video.

You can use any multi-label prediction, action recognition, and visual classification networks to extract those features. We believe this will be easier and barely impact performance.

Thank you for your detailed reply! I really appreciate it. I also want to ask that how many local key frames do you get from a video and how you sample the local key frame(s) from lots of frames in a single video. Sorry to disturb you. Thanks a lot!

For the Local-Keyframe Encoder, we select 30 keyframes evenly in the time sequence for each video.

So for every selected keyframe $f_t$ , the input of the Short-range Encoder is 20 frames ( $f_{t-10}$ to $f_{t+10}$ ) or just 2 frames ( $f_{t-10}$ and $f_{t+10}$ )? I have the same question for the Long-range Encoder. And if you select 30 keyframes evenly, should the sequence range of uniform sampling be smaller than the whole video sequence range? Otherwise, I'm afraid that $f_{t-n}$ and $f_{t+n}$ may go beyond the video frame range:slightly_smiling_face:.

[ylqi]

We selected fewer key-frames for Long-range and Short-range encoders. Please make sure the $n$ is smaller than the minimized uniform sampling rate. Since 3D CNNs need a certain number of inputs, the former is correct.