ViTPose for Human Mesh Recovery

[jebastin-nadar]

Use the detected 2D keypoints from ViTPose and save fitting results:

Could you please which ViTPose model was used to generate the keypoints for HMR? Also would it be possible to publicly provide the generated keypoints which can be used to test other methods for HMR?

[moonbow721]

Hi, thanks for your great question!

For the 2D keypoint detection in our paper, we used the whole-body version of ViTPose. We chose this model because it allowed us to use a consistent set of keypoints as supervision for both the standard HMR and the whole-body HMR tasks.

Specifically, the model we used is:

• Config file: https://github.com/ViTAE-Transformer/ViTPose/blob/main/configs/wholebody/2d_kpt_sview_rgb_img/topdown_heatmap/coco-wholebody/ViTPose_huge_wholebody_256x192.py
• Model weights: https://1drv.ms/u/s!AimBgYV7JjTlgccoXv8rCUgVe7oD9Q?e=ZBw6gR)](https://1drv.ms/u/s!AimBgYV7JjTlgccoXv8rCUgVe7oD9Q?e=ZBw6gR

Regarding your request to provide the pre-processed keypoints, that's an excellent suggestion for helping others reproduce and compare results. Unfortunately, due to personal reasons, I am currently away from my university and do not have access to the server with the data from our experiments.

I won't be able to upload them right away, but I should be able to provide them in about two weeks. I will update this thread as soon as they are available.

Thank you for your understanding!

[Queequeg92]

@jebastin-nadar rtmlib is recommended to detect 2d keypoints. It's very easy to use.
https://github.com/Tau-J/rtmlib?tab=readme-ov-file

[jebastin-nadar]

Hi @moonbow721 ! Would it be possible to share the keypoints? I generated mine, and got a score of 56.12 for HMR, which is close to the number reported in the paper (56.05). But I had issues with installing mmdet, so I used a different object detector, before getting vitpose keypoints. I think if you can share the keypoints, I should be able to reproduce the number exactly. Thanks for the help!

[moonbow721]

Hi @jebastin-nadar,

Sorry for the late reply. I have just uploaded the keypoint files used in our experiments:
https://drive.google.com/file/d/1WjQ8tIixuhAvh7msUX04FeQ_fF-FKahw/view?usp=drive_link

However, please note that the exact number can be influenced by factors like your environment.

In addition, the HMR result in the paper was actually from an older checkpoint, which was trained before we started the whole-body project. If you want to reproduce that specific number, you can also try using this checkpoint:
https://drive.google.com/file/d/19RO_6qMLUKxVndPolFjhEreTyoYhcOes/view?usp=drive_link

Hope this helps!