Multi-modal point cloud completion, which utilizes a complete image and a partial point cloud as input, is a crucial task in 3D computer vision. Previous methods commonly employ a cross-attention mechanism to fuse point clouds and images. However, these approaches often fail to fully leverage image information and overlook the intrinsic geometric details of point clouds that could complement the image modality. To address these challenges, we propose an interleaved attention enhanced Transformer (IAET) with three main components, i.e., token embedding, bidirectional token supplement, and coarse-to-fine decoding. IAET incorporates a novel interleaved attention mechanism to enable bidirectional information supplementation between the point cloud and image modalities. Additionally, to maximize the use of the supplemented image information, we introduce a view-guided upsampling module that leverages image tokens as queries to guide the generation of detailed point cloud structures. Extensive experiments demonstrate the effectiveness of IAET, highlighting its state-of-the-art performance on multi-modal point cloud completion benchmarks in various scenarios.
Use the code in dataloader.py to load the dataset.
First, please download the ShapeNetViPC-Dataset (143GB, code: ar8l). Then run cat ShapeNetViPC-Dataset.tar.gz* | tar zx, you will get ShapeNetViPC-Dataset contains three folders: ShapeNetViPC-Partial, ShapeNetViPC-GT, and ShapeNetViPC-View.
For each object, the dataset includes partial point clouds (ShapeNetViPC-Patial), complete point clouds (ShapeNetViPC-GT), and corresponding images (ShapeNetViPC-View) from 24 different views.
The KITTI dataset used in this work is sourced from the Cross-PCC.
Notably, You need to replace the current paths in the following files with the absolute path to your dataset: config_3depn.py:8, config_vipc.py:12, and eval_vipc.py:21.
- Ubuntu: 18.04 and above
- CUDA: 11.3 and above
- PyTorch: 1.10.1 and above
activate your environment and then
cd cuda/ChamferDistance
python setup.py install
and
cd cuda/pointnet2_ops_lib
python setup.py install
The file config_vipc.py and config_3depn.py contain the configuration for all the training parameters.
Our code supports multi-GPU parallel training. Training on ViPC dataset with single GPU:
CUDA_VISIBLE_DEVICES=0 python train_vipc.py
Training on ViPC dataset with multiple GPUs (e.g., 4):
CUDA_VISIBLE_DEVICES=0, 1, 2, 3 python train_vipc.py
Training on 3DEPN dataset with single GPU:
CUDA_VISIBLE_DEVICES=0 python train_3depn.py
To evaluate the models (select the specific category in config_vipc.py):
CUDA_VISIBLE_DEVICES=0 python eval_vipc.py
Some of the code of this repo is borrowed from:
@inproceedings{fang2025iaet,
title={Multi-Modal Point Cloud Completion with Interleaved Attention Enhanced Transformer},
author={Chenghao Fang and Jianqing Liang and Jiye Liang and Hangkun Wang and Kaixuan Yao and Feilong Cao},
booktitle={Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence (IJCAI)},
pages={963-971},
year={2025}
}This project is open sourced under MIT license.