Code repository for the paper: Reconstructing Hands in 3D with Transformers

Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, Jitendra Malik

• [2024/06] HaMeR received the 2nd place award in the Ego-Pose Hands task of the Ego-Exo4D Challenge! Please check the validation report.
• [2024/05] We have released the evaluation pipeline!
• [2024/05] We have released the HInt dataset annotations! Please check here.
• [2023/12] Original release!

First you need to clone the repo:

git clone https://github.com/Arthur151/hamer.git
cd hamer

Create a virtual environment for HaMeR via conda:

conda create --name hamer python=3.10
conda activate hamer

Then, you can install the rest of the dependencies. This is for CUDA 11.7, but you can adapt accordingly:

# replace with the cuda version you use
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e .[all]
pip install -v -e ViTPose

You also need to download the trained models:

bash fetch_demo_data.sh
cp /home/yu/data/afv_data/smpl_model_data/mano/MANO_RIGHT.pkl _DATA/data/mano/

Continue with the installation steps:

bash fetch_demo_data.sh

python demo.py \
    --img_folder example_data --out_folder demo_out \
    --batch_size=48 --side_view --save_mesh --full_frame

We have released the annotations for the HInt dataset. Please follow the instructions here

First, download the training data to ./hamer_training_data/ by running:

bash fetch_training_data.sh

Then you can start training using the following command:

python train.py exp_name=hamer data=mix_all experiment=hamer_vit_transformer trainer=gpu launcher=local

Checkpoints and logs will be saved to ./logs/.

Download the evaluation metadata to ./hamer_evaluation_data/. Additionally, download the FreiHAND, HO-3D, and HInt dataset images and update the corresponding paths in hamer/configs/datasets_eval.yaml.

Run evaluation on multiple datasets as follows, results are stored in results/eval_regression.csv.

python eval.py --dataset 'FREIHAND-VAL,HO3D-VAL,NEWDAYS-TEST-ALL,NEWDAYS-TEST-VIS,NEWDAYS-TEST-OCC,EPICK-TEST-ALL,EPICK-TEST-VIS,EPICK-TEST-OCC,EGO4D-TEST-ALL,EGO4D-TEST-VIS,EGO4D-TEST-OCC'

Results for HInt are stored in results/eval_regression.csv. For FreiHAND and HO-3D you get as output a .json file that can be used for evaluation using their corresponding evaluation processes.

Parts of the code are taken or adapted from the following repos:

• 4DHumans
• SLAHMR
• ProHMR
• SPIN
• SMPLify-X
• HMR
• ViTPose
• Detectron2

Additionally, we thank StabilityAI for a generous compute grant that enabled this work.

• Wentao Hu integrated the hand parameters predicted by HaMeR into SMPL-X - Mano2Smpl-X

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{pavlakos2024reconstructing,
    title={Reconstructing Hands in 3{D} with Transformers},
    author={Pavlakos, Georgios and Shan, Dandan and Radosavovic, Ilija and Kanazawa, Angjoo and Fouhey, David and Malik, Jitendra},
    booktitle={CVPR},
    year={2024}
}