This is the official PyTorch implementation of our AAAI 2025 paper "PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Spatio-Temporal State Space Model".
The project is developed under the following environment:
- Python 3.8.5
- PyTorch 1.13.1+cu117
- torchvision 0.14.1+cu117
- torchaudio 0.13.1+cu117
- CUDA 11.7
For installation of the project dependencies, please run:
conda create -n posemamba python=3.8.5
conda activate posemamba
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
cd kernels/selective_scan && pip install -e .
- Download the fine-tuned Stacked Hourglass detections of MotionBERT's preprocessed H3.6M data here and unzip it to 'data/motion3d', or direct download our processed data here and unzip it.
- Slice the motion clips by running the following python code in
tools/convert_h36m.py:
python convert_h36m.py
Please refer to - MotionAGFormer for dataset setup.
After dataset preparation, you can train the model as follows:
You can train Human3.6M with the following command:
CUDA_VISIBLE_DEVICES=0 python train.py --config <PATH-TO-CONFIG> --checkpoint <PATH-TO-CHECKPOINT>
where config files are located at configs/h36m.
Please refer to - MotionAGFormer for training.
We provide checkpoint. You can download and unzip it to get pretrained weight.
| Method | frames | Params | MACs | Human3.6M weights |
|---|---|---|---|---|
| PoseMamba-S | 243 | 0.9M | 3.6G | PoseMamba-S |
| PoseMamba-B | 243 | 3.4M | 13.9G | PoseMamba-B |
| PoseMamba-L | 243 | 6.7M | 27.9G | PoseMamba-L |
After downloading the weight, you can evaluate Human3.6M models by:
python train.py --eval-only --checkpoint <CHECKPOINT-DIRECTORY> --checkpoint-file <CHECKPOINT-FILE-NAME> --config <PATH-TO-CONFIG>
For example if PoseMamba-L of H.36M is downloaded and put in checkpoint directory, then we can run:
python train.py --eval-only --checkpoint checkpoint --checkpoint-file PoseMamba-l-h36m.pth.tr --config configs/h36m/PoseMamba-large.yaml
Our demo is a modified version of the one provided by MotionAGFormer repository. First, you need to download YOLOv3 and HRNet pretrained models here and put it in the './demo/lib/checkpoint' directory. Next, download our base model checkpoint from here and put it in the './checkpoint' directory. Then, you need to put your in-the-wild videos in the './demo/video' directory. We provide demo. You can download and unzip it to get demo file. Run the command below:
python vis.py --video sample_video.mp4 --gpu 0
Sample demo output:
Our code refers to the following repositories:
We thank the authors for releasing their codes.
If you find our work useful for your project, please consider citing the paper:
@article{huang2024posemamba,
title={PoseMamba: Monocular 3D Human Pose Estimation with Bidirectional Global-Local Spatio-Temporal State Space Model},
author={Huang, Yunlong and Liu, Junshuo and Xian, Ke and Qiu, Robert Caiming},
journal={arXiv preprint arXiv:2408.03540},
year={2024}
}