Surface Keypoint Representation for Multi-Object and
Articulated Human–Object Interaction Generation

Xiaogang Peng¹  Zeyu Han¹  Zichong Meng¹  Yiming Xie¹  Jihua Zhu²  Gang Hua³  Huaizu Jiang^1
1Northeastern University ²Xi'an Jiaotong University ³Amazon

Official PyTorch implementation of "Surface Keypoint Representation for Multi-Object and Articulated Human–Object Interaction Generation."

• [2026-08-05] 🚀 Initial training/sampling code released.

• Autoencoder (CausalTripleAE) training + reconstruction eval
• Three-stage diffusion training (object / contact / human)
• Multi-stage sampling + marker→SMPL-X fitting
• Release data-prep for Parahome and OMOMO datasets
• Release data-prep for HIMO and ARCTIC datasets
• Release pretrained checkpoints
• Evaluation scripts

Staged, latent-space generation of multi-object and articulated human–object interactions from text, built on a surface-keypoint representation. The pipeline compresses human motion, object motion, and the contact distance field into a frozen Causal Autoencoder (CausalTripleAE) latent space, then generates in three cascaded diffusion stages:

Each stage is a separate checkpoint that shares the one frozen autoencoder.

models/
  casual_ae.py      # CausalTripleAE — frozen human/object/contact latent space
  stage1_omd.py     # Stage1_OMD  (OaP-Object)  — object motion diffusion
  stage2_cdd.py     # Stage2_CDD  (OaP-Contact) — contact distance-field diffusion
  stage3_mmd.py     # Stage3_MMD  (OaP-Multi)   — human motion diffusion
  fitting_model.py  # marker -> SMPL-X regressor + test-time optimization
  diffusions/       # flow-matching (SiT/transport) + Gaussian-diffusion backends
datasets/
  __init__.py       # DataModule: routes --data_name to a dataset + collate_fn
  dataset_marker.py # main HOI marker dataset  (--data_name marker_HOI)
  tensors.py        # collate functions
tools/
  train_ae.py  train_diffusion.py  train_fitter.py        # training entry points
  compute_contact_stats.py  compute_baseline_stats.py      # normalization stats
  eval_ae.py  eval_diffusion.py                            # evaluation
  multi_stage_sample.py  sample_ae.py  sample_fitter.py    # sampling
  render_utils.py  render_all_results.py  render_dataset.py # rendering

conda activate skhoi
export PATH=/shared/centos7/cuda/11.8/bin:$PATH
export LD_LIBRARY_PATH=/shared/centos7/cuda/11.8/lib64/:$LD_LIBRARY_PATH

• Data lives under ./data; training uses --dataset {parahome,omomo,arctic,himo,all}.
• Marker representation: --num_joints 138 --num_obj_verts 9 (138 body markers, 3 object keypoints × 3).
• Third-party libs (torchdiffeq, human_body_prior) and body models (body_models/) are not committed — install / place them separately.

All commands below are single-node python invocations; adapt them to your own cluster scheduler (SLURM / PBS / …) as needed.

Trains the frozen latent space for all three modalities (human + object + per-marker contact). Use --context_len 0 --pred_len 0 so motion and contact are the full, aligned sequence.

python tools/train_ae.py --data_name marker_HOI \
    --model_arc CausalTripleAE --pose_rep 2d_joints \
    --exp_name CausalTripleAE \
    --context_len 0 --pred_len 0 --num_frames 64 --batch_size 8 --lr 1e-4 \
    --num_joints 138 --num_obj_verts 9 --dataset all \
    --predict_contact_map --contact_type field \
    --contact_mode per_marker --contact_loss smoothl1 --contact_pos_weight 5 --ae_branches all \
    --save_interval 1000 --eval_interval 1000 \
    --save_dir ./checkpoints
    --overwrite

Evaluate reconstruction quality (human MPJPE / object vertex error / contact P·R·F1):

python tools/eval_ae.py --data_name marker_HOI --model_arc CausalTripleAE \
    --pose_rep 2d_joints --context_len 0 --pred_len 0 --num_frames 64 \
    --num_joints 138 --num_obj_verts 9 --dataset all --contact_type field --predict_contact_map \
    --batch_size 64 --model_path ./checkpoints/CausalTripleAE/best_model.pt

The Stage-2 diffusion standardizes the contact latent. Compute the stats once and save them next to the AE checkpoint (contact_latent_stats.pt):

python tools/compute_contact_stats.py --data_name marker_HOI --model_arc CausalTripleAE \
    --pose_rep 2d_joints --context_len 0 --pred_len 0 --num_frames 64 \
    --num_joints 138 --num_obj_verts 9 --dataset all --contact_type field --predict_contact_map \
    --batch_size 32 --model_path ./checkpoints/CausalTripleAE/best_model.pt

The diffusion stages load the frozen AE from $TRIPLE_AE_CKPT (default: ./checkpoints/CausalTripleAE/best_model.pt). Override if needed:

export TRIPLE_AE_CKPT=./checkpoints/CausalTripleAE/best_model.pt

python tools/train_diffusion.py --data_name marker_HOI \
    --model_arc Stage1 --pose_rep 2d_joints --exp_name stage1_object \
    --context_len 4 --pred_len 120 --batch_size 64 --lr 1e-4 \
    --num_joints 138 --num_obj_verts 9 --dataset parahome --use_ae --add_human_init \
    --save_dir ./checkpoints --overwrite

Conditions on the (GT or Stage-1) object latent; predicts the per-marker contact field.

python tools/train_diffusion.py --data_name marker_HOI \
    --model_arc Stage2 --pose_rep 2d_joints --exp_name stage2_contact \
    --context_len 4 --pred_len 120 --batch_size 64 --lr 1e-4 \
    --num_joints 138 --num_obj_verts 9 --dataset parahome --use_ae \
    --predict_contact_map --contact_type field \
    --save_dir ./checkpoints/ --overwrite

Conditions on the object motion (--cond_obj_motion); predicts full-body human motion.

python tools/train_diffusion.py --data_name marker_HOI \
    --model_arc Stage3 --pose_rep 2d_joints --exp_name stage3_human \
    --context_len 4 --pred_len 120 --batch_size 64 --lr 1e-4 \
    --num_joints 138 --num_obj_verts 9 --dataset parahome --use_ae \
    --predict_contact_map --cond_obj_motion --use_obj_feat --add_human_init \
    --save_dir ./checkpoints/ --overwrite

Swap --dataset parahome for omomo / arctic / himo to train on other benchmarks. Best models are saved as best_model.pt; periodic checkpoints as model000XXXXXX.pt.

tools/multi_stage_sample.py runs the cascade and decodes back to world coordinates. Per-stage checkpoints are set via the STAGE1_CKPT / STAGE2_CKPT / STAGE3_CKPT env vars (see the script header). --stage selects how much of the cascade to run:

python tools/multi_stage_sample.py --data_name marker_HOI --pose_rep 2d_joints \
    --context_len 4 --pred_len 120 --num_samples 10 --num_joints 138 --num_obj_verts 9 \
    --dataset parahome --use_ae --stage three --cond_obj_motion --add_human_init \
    --num_opt_steps 0 --vis_smplx

--stage options: one (object only), two / two_gt (contact from predicted / GT object), three / three_gt (human from predicted / GT object).

--num_opt_steps controls the Stage-3 contact-guided test-time optimization: after the human diffusion samples an initial motion, the result is refined for num_opt_steps iterations to better satisfy the predicted contact constraints. 0 disables it (fast, raw diffusion output); a positive value (e.g. 200) runs the refinement, trading extra inference time for tighter human–object contact. It only applies to --stage three / three_gt.

If you find this work useful, please consider citing:

@misc{peng2026surfacekeypointrepresentationmultiobject,
    title={Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation}, 
    author={Xiaogang Peng and Zeyu Han and Zichong Meng and Yiming Xie and Jihua Zhu and Gang Hua and Huaizu Jiang},
    year={2026},
    eprint={2608.03158},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2608.03158}, 
}

This project builds on ideas and code from prior work, including CHOIS, HOI-Diff, HIMO, and the SiT / flow-matching and diffusion-transformer literature. We thank the authors of the ParaHome, OMOMO, ARCTIC, and HIMO datasets.

This codebase is released under the MIT LICENSE. Please note that it also relies on external libraries and datasets, each of which may be subject to their own licenses and terms of use