How to use pred_keypoints_3d

[atonalfreerider]

Thank you for creating this package. I am able to run my own script and get hand data.

I am graphing my own hands in Unity and I am saving the WiLoR output to a json. I have these values for each hand from wilor_preds:

'global_orient' (1,1,3)
'hand_pose' (1,15,3)
'betas' (1,10)
'pred_cam' (1,3)
'pred_keypoints_3d' (1,21,3)
'pred_vertices' (1,778,3)
'pred_cam_t_full' (1,3)
'scaled_focal_length' float64
'pred_keypoints_2d (1,21,2)

The value pred_keypoints_3d gives me points that I can use to draw a 3d hand in Unity, but the points are fixed near the origin. I can't tell if global_orient is an Euler rotation or how to use it. Do the hands get translated relative to the camera via pred_cam_t_full?

[PannagaS]

@atonalfreerider yes even I am having similar doubts. Like pred_cam_t_full appears to be camera translation vector wrt wrist (I'm assuming), but the values I am getting make no sense tbh. They are very off from the true value I measure.

[warmshao]

Hi @atonalfreerider,

Thank you for your question and for using the WiLoR-mini package. It seems like you're on the right track with visualizing the hand data in Unity.

Regarding your queries:

1. Global Orientation: The global_orient is likely an Euler rotation. You might need to convert it to a quaternion or rotation matrix to apply it correctly in Unity.

2. Camera Translation: The pred_cam_t_full appears to be a translation vector that positions the hand relative to the camera. You can use this vector to translate the hand model in your 3D scene.

I hope this helps! If you have further questions or need more clarification, feel free to ask.

Best regards,
RepoAssistant

[BaldPulse]

Here's a late reply.

I assume you want your 3d key points in the camera's coordinate frame, which you can find more information about here. You will only need pred_cam_t_full to get that. The reason people are getting weird pred_cam_t_full values is because the pipeline object requires an accurate focal length value to estimate translation and currently the value is hard-coded to be 5000. I have submitted a PR to illustrate how to set a custom focal length. You can refer to this pr for more details.