R3DP: Real-Time 3D-Aware Policy for
Embodied Manipulation

Yuhao Zhang^1,2*, Wanxi Dong^3,4*, Yue Shi^1*, Yi Liang^1*, Jingnan Gao¹, Qiaochu Yang^1,4, Yaxing Lyu⁷, Zhixuan Liang⁵,
Yibin Liu⁸, Congsheng Xu¹, Xianda Guo^6,2, Wei Sui², Yaohui Jin¹, Xiaokang Yang¹, Yanyan Xu¹, Yao Mu^1†

¹Shanghai Jiao Tong University  ²D-Robotics  ³Southern University of Science and Technology  ⁴Xspark AI
⁵The University of Hong Kong  ⁶Wuhan University  ⁷Xiamen University Malaysia  ⁸Northeastern University

^*Equal Contribution  ^†Corresponding Author

R3DP (Real-time 3D-aware Policy) offers a plug-and-play solution that injects the 3D priors of large-scale foundation models (e.g., VGGT) into imitation-learning policies such as Diffusion Policy — without sacrificing real-time control. It endows RGB-only policies with strong spatial understanding while preserving the inherent strengths and speed of 2D policies, requiring no depth sensors or point-cloud rigs.

R3DP runs an asynchronous fast–slow collaboration: the slow branch queries the heavy 3D foundation model (VGGT) only on sparse key frames every τ steps, while a lightweight Temporal Feature Prediction Network (TFPNet) — distilled from VGGT and conditioned on historical frames — fills in 3D-aware features for all intermediate frames in real time. A Multi-View Feature Fuser (MVFF) then aggregates per-view features using camera intrinsics and extrinsics via cross-attention with PRoPE-style projective encoding, producing a spatially consistent representation that conditions the diffusion policy head.

R3DP runs inside the RoboTwin simulation environment. Installation has two parts: (1) the base RoboTwin environment, and (2) the R3DP policy and its 3D backbone.

Follow INSTALLATION.md for the full RoboTwin setup (conda env, SAPIEN, mplib patches, and asset download). In short:

conda create -n r3dp python=3.8
conda activate r3dp

pip install torch==2.4.1 torchvision sapien==3.0.0b1 scipy==1.10.1 mplib==0.1.1 \
    gymnasium==0.29.1 trimesh==4.4.3 open3d==0.18.0 imageio==2.34.2 pydantic \
    zarr openai huggingface_hub==0.25.0

cd third_party/pytorch3d_simplified && pip install -e . && cd ../..

# Download simulation assets
python ./script/download_asset.py
unzip aloha_urdf.zip && unzip main_models.zip

⚠️ Don't forget the mplib library patches described in INSTALLATION.md — they are required for the planner to work.

cd policy/R3DP

cd vggt && pip install -e . && cd ..

pip install -e .

pip install wandb hydra-core==1.2.0 diffusers==0.11.1 einops accelerate timm

(a) Foundation weights:

(b) R3DP weights:

Generate expert demonstrations for a task using RoboTwin's automated collectors:

bash run_task.sh ${task_name} ${gpu_id}
# e.g.
bash run_task.sh block_hammer_beat 0

cd policy/R3DP
bash train.sh ${task_name} ${head_camera_type} ${expert_data_num} ${seed} ${gpu_ids}
# e.g. train on 4 GPUs
bash train.sh block_hammer_beat L515 100 0 0,1,2,3

The script automatically converts .pkl demonstrations into a .zarr dataset (via script/pkl2zarr_r3dp.py) if it does not already exist, then launches multi-GPU training with accelerate. Checkpoints are written to policy/R3DP/checkpoints/.

cd policy/R3DP
bash eval_r3dp.sh ${task_name} ${head_camera_type} ${expert_data_num} ${checkpoint_num} ${seed} ${gpu_id} ${tau}
# e.g.
bash eval_r3dp.sh block_hammer_beat L515 100 100 0 0 4

Here tau (τ) controls the fast–slow schedule — how often the slow VGGT branch is queried. Larger τ → faster inference; smaller τ → slightly higher accuracy. R3DP is robust across a wide range of τ.

R3DP is built upon several excellent open-source projects:

• RoboTwin — dual-arm manipulation benchmark and simulation environment.
• VGGT — the 3D vision foundation model used as our slow branch.
• Diffusion Policy — the base visuomotor policy.
• DINOv2 — visual backbone for TFPNet.

We thank the authors for making their work publicly available.

If you find R3DP useful in your research, please consider citing:

@misc{zhang2026r3dprealtime3dawarepolicy,
      title={R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation}, 
      author={Yuhao Zhang and Wanxi Dong and Yue Shi and Yi Liang and Jingnan Gao and Qiaochu Yang and Yaxing Lyu and Zhixuan Liang and Yibin Liu and Congsheng Xu and Xianda Guo and Wei Sui and Yaohui Jin and Xiaokang Yang and Yanyan Xu and Yao Mu},
      year={2026},
      eprint={2603.14498},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.14498}, 
}