This repository contains the official implementation of D-Prism: Differentiable Primitives for Structured Dynamic Modeling (CVPR 2026).

D-Prism is the first framework to achieve high-fidelity structured dynamic modeling by extending differentiable primitives to the dynamic domain. Specifically, we bind 3DGS to primitive surfaces, leveraging their respective strengths in appearance and geometry. We introduce a deformation network to control primitive motion, ensuring it accurately matches the object’s movement. Furthermore, we design a novel adaptive control strategy to dynamically adjust primitive counts, better matching objects’ true spatial footprint.

• Training/inference code
• Dataset release (Dynamic Primitive Dataset and real-world captured datasets)

# Create conda environment
conda create -n d-prism python=3.9
conda activate d-prism
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

# Install nvdiffrast
pip install git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch
pip install git+https://github.com/NVlabs/nvdiffrast/

# Install pytorch3d
export FORCE_CUDA=1
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

# Clone this repository
git clone https://github.com/zju3dv/d-prism.git
cd d-prism

# Install submodules
pip install dprism/submodules/diff-gaussian-rasterization
pip install dprism/submodules/simple-knn

# Install other dependencies
pip install -r requirements.txt

Create a data/ directory in the root of the project to store the datasets:

mkdir data

• Description: Contains 6 structured dynamic scenes (Rubik's Cube, Treasure Box, Door, Pliers, Folding Chair, and Sunglasses).
• Download: Download from Hugging Face.
• Setup:
  • Rubik's Cube: Place the dataset folder under data/rubiks_cube/ (default: data/rubiks_cube/rubiks_cube_seq_EASY_large).
  • Other scenes (PartNet-Mobility): Place the dataset folders under data/partnet-mobility-v0/ (e.g., data/partnet-mobility-v0/100221_new5 for the Treasure Box scene).

• Download: Download the D-NeRF Dataset Zip.
• Setup: Extract the zip archive directly under data/d-nerf/ so that the structure matches the default configurations:

  data/d-nerf/
  ├── jumpingjacks/
  ├── mutant/
  └── ...
  

• Download: Download from Hugging Face.
• Setup: Place the robotic hand/arm dataset folder under data/primitive_real/ (default: data/primitive_real/robotic_hand_moving3_release).

• Download: Download the dataset from the Shape of Motion Google Drive Link.
• Setup: Place the Dycheck iPhone dataset folder under data/iPhone_som/ (default: data/iPhone_som/haru-sit).

• Download: Refer to the dataset download method provided by DG-Mesh. The dataset files can be downloaded from the DG-Mesh Releases page.
• Setup: Place the record3d/iphone dataset folder under data/iphone-captured/ (default: data/iphone-captured/tiger).

To train on the Rubik's Cube scene:

python dprism/train.py --config dprism/configs/rubiks_cube/easy_large.yaml

To train on other scenes (e.g., Treasure Box):

python dprism/train.py --config dprism/configs/partnet-mobility-v0/100221.yaml

To train on D-NeRF sequences (e.g., jumpingjacks):

python dprism/train.py --config dprism/configs/d-nerf/jumpingjacks_pri.yaml

To train on real-world robotic hand/arm sequences:

python dprism/train.py --config dprism/configs/primitive_real/robotic_hand_moving3.yaml

To train on Dycheck iPhone sequences:

python dprism/train.py --config dprism/configs/iphone_som/haru-sit_pri.yaml

To train on record3d/iphone sequences:

python dprism/train.py --config dprism/configs/iphone/tiger_pri.yaml

After the initial training, you can refine the trained models using the refinement script.

# Example for Rubik's Cube scene
python dprism/train_refine.py --config dprism/configs/rubiks_cube/easy_large_refine.yaml

# Example for PartNet-Mobility scenes (e.g., Treasure Box(100221))
python dprism/train_refine.py --config dprism/configs/partnet-mobility-v0/100221_refine.yaml

# Example for D-NeRF scenes (e.g., Stand-Up)
python dprism/train_refine.py --config dprism/configs/d-nerf/standup_pri_refine.yaml

We evaluate both Structured Dynamic Reconstruction and Structured Motion Tracking in a unified way using dprism/eval_3d_tracking.py. It computes:

• Structured Dynamic Reconstruction: Dynamic Chamfer Distance ($CD_d$) and Dynamic Earth Mover's Distance ($EMD_d$) between the reconstructed dynamic primitives and the ground-truth meshes in dynamic settings.
• Structured Motion Tracking: 3D End-Point Error (EPE) and tracking accuracy ($\delta_{3D}^{0.05}$ and $\delta_{3D}^{0.10}$) for the predicted motion of sampled points.

Before running the evaluation, you must compile the StructuralLoss library:

cd dprism/metrics/pytorch_structural_losses/
make clean
make
cd ../../../

First, run the ground-truth generation script. This samples 50,000 points from the canonical mesh, applies barycentric coordinates, and saves the Ground Truth 3D tracking data (gt_tracking_T_N_3.npy) under the dataset source directory:

# Example for Rubik's Cube scene
python dprism/gen_3d_tracking_gt.py --config dprism/configs/rubiks_cube/easy_large_refine_eval_3d.yaml

# Example for PartNet-Mobility scenes (e.g., Treasure Box(100221))
python dprism/gen_3d_tracking_gt.py --config dprism/configs/partnet-mobility-v0/100221_refine_eval_3d.yaml

Run the tracking evaluation script. It calculates the EPE, CD, and EMD accuracy metrics, saving the results (flow_3d_acc.txt and pts_3d_acc.txt) to the output directory:

# Example for Rubik's Cube scene
python dprism/eval_3d_tracking.py --config dprism/configs/rubiks_cube/easy_large_refine_eval_3d.yaml

# Example for PartNet-Mobility scenes (e.g., Treasure Box(100221))
python dprism/eval_3d_tracking.py --config dprism/configs/partnet-mobility-v0/100221_refine_eval_3d.yaml

If you find our code or paper helpful, please consider citing:

@inproceedings{yu2026dprism,
  title     = {D-Prism: Differentiable Primitives for Structured Dynamic Modeling},
  author    = {Yu, Xingyuan and Li, Yijin and Zeng, Chong and Ming, Yuhang and Bao, Hujun and Zhang, Guofeng},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

We thank the authors of Deformable 3DGS, 3DGS, and DG-Mesh for their excellent codebases.