Video tracking

[emmanueliarussi]

Hi! thanks for the great repo and for sharing code/weights!

I’m fitting a custom video by generating per-frame keypoints with ViTPose and then running run.tester.wholebody.demo_fit per frame. It works, but results are very jittery. The denoiser script seems tailored to bundled "standard" datasets rather than arbitrary frames.

Questions:

1. What’s the recommended pipeline for custom videos (keypoints → smoothing → fitting)?

2. Can the denoiser/temporal smoother run on external ViTPose JSON? An example command/config would help.

3. Is there a built-in script to process a whole folder/video with temporal consistency, or flags in demo_fit to stabilize?

[moonbow721]

Hi! Thanks for the great questions and for trying our code on your custom videos.

You've encountered an important point. Our pose prior model, DPoser-X, was primarily designed to constrain the pose for a single frame. This is why you are seeing jittery results when you process video frames independently.

The jitter is caused by a combination of factors:

1. The inherent randomness in the diffusion sampling and optimization process.
2. Potential instabilities or slight variations in the 2D keypoints estimated by ViTPose from frame to frame.

When these factors are applied independently to each frame, the resulting SMPL parameters will naturally lack temporal smoothness.

Here are the answers to your specific questions:

1. What’s the recommended pipeline for custom videos?

For the best results on a custom video, I would recommend the following pipeline:

• Keypoint Detection: In our later tests, we found that RTMPose (from MMPose) seems to provide more stable whole-body keypoint estimations compared to ViTPose. You might get better base data by switching to it.
• 2D Keypoint Smoothing: Apply a temporal smoothing filter like a Gaussian filter (see code in lib/utils/misc.py) to the detected 2D keypoints before passing them to the fitting stage. This will provide a much more stable input.
• Temporal Fitting: Modify the fitting script to process the video as a whole or in chunks of consecutive frames. You can then add a temporal smoothing term to the loss function across the batch dimension. You can refer to our run/tester/body/motion_denoising script for an example of how to implement such a smoothing term.

2. Can the denoiser/temporal smoother run on external ViTPose JSON?

Yes, absolutely. The run/tester/wholebody/demo_fit.py script is already set up to handle JSON files from several common 2D pose estimators, including 'mmpose', 'vitpose', and 'openpose'.

You can see the code that handles this here:

DPoser-X/run/tester/wholebody/demo_fit.py

Lines 60 to 73 in 1829905

	if args.kpts == 'openpose':
	keypoints = np.array(json_data['people'][0]['pose_keypoints_2d'] +
	json_data['people'][0]['hand_left_keypoints_2d'] +
	json_data['people'][0]['hand_right_keypoints_2d'] +
	json_data['people'][0]['face_keypoints_2d'][:204]).reshape((135, 3))
	elif args.kpts == 'mmpose':
	mm_keypoints = np.array(json_data[0]['keypoints'])
	keypoint_scores = np.array(json_data[0]['keypoint_scores'])
	keypoints = mmpose_to_openpose(mm_keypoints, keypoint_scores)
	elif args.kpts == 'vitpose':
	vit_keypoints = np.array(json_data[0]['keypoints'])
	keypoints = vitpose_to_openpose(vit_keypoints)
	else:
	raise NotImplementedError

3. Is there a built-in script to process a whole folder/video with temporal consistency?

Currently, the demo_fit.py script does not have a built-in temporal smoother and processes each frame with a batch size of 1 by default.

However, the underlying smplify.py file that it calls does support an arbitrary batch size. Therefore, it is entirely feasible to modify the demo_fit.py script to read multiple frames at once (as a batch) and then add a smoothing loss term across that batch dimension in smplify.py to enforce temporal consistency.

I hope this detailed explanation helps you achieve smoother results with your videos!

[emmanueliarussi]

Thanks for the detailed reply @moonbow721 ! I’ll give your suggestions a try and share the script if I manage to get consistent results on custom videos.