CenterLPS: Segment Instances by Centers for LiDAR Panoptic Segmentation

[Paper]

• easydict
• numba
• numpy
• pyyaml
• python=3.7
• scikit-learn
• scipy
• spconv=1.1
• tensorboard=2.7.0
• torch=1.6
• torchvision=0.7.0
• torch-cluster=1.6.0
• torch-scatter=2.0.8
• tqdm

Please download the SemanticKITTI dataset and nuScene to the folder data and the structure of the folder should look like:

./
├── 
├── ...
└── data/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |       ├── 000000.label
        |       ├── 000001.label
        |       └── ...
        ├── 08/ # for validation
        ├── 11/ # 11-21 for testing
        └── 21/
	        └── ...
    ├──nuscenes
        ├── lidarseg/           
        ├── maps
        ├── panoptic
        └── ...

The training pipeline of our CenterLPS consists of three steps: 1) semantic segmentation training following GASN; 2) offset training; The first step give us the semantic backbone. Then we train the offset head to provide instance offsets. 3) instance training. The second step provide the semantic backbone with the offset head. Then our center-based instance segmentation and mask fusion are performed to obtain the final panoptic segmentation. We provide the corresponding pretrained model of each step. The training and inferencing details are further explained in this section.

All the slurm and pytorch version of scripts are tested and should work well.

We put all pretrained models in the folder weights.

We provide inferencing scripts for our CenterLPS.

Before inferencing with our full model CenterLPS, remember to use the pretrained model (of step 3) in the folder weights or use the model trained by yourself (in step 3) and make sure you pass the right path to --pretrained_ckpt option. For validation, you can use the following command.

To evaluate on SemanticKITTI validation:

$ python cfg_train.py --cfg cfgs/gcd_center_aware.yaml --onlyval --pretrained_ckpt weights/kitti_aware.pth

To evaluate on nuScenes validation:

$ python cfg_train.py --cfg cfgs/nuscenes/gcd_center_aware.yaml --onlyval --pretrained_ckpt weights/nuscenes_aware.pth

The training code for the semantic backbone has not been incorporated yet. For now, please train the semantic backbone using the code released by GASN or use the provided pretrained model of step 1.

The training scripts could be found in ./scripts/*_train.sh. Before using the training scripts, please use the pretrained model (of step 1) in the folder weights or put the model trained (in step 1) to weights. For training with a single GPU, you can also use the following command.

For SemanticKITTI:

$ python cfg_train.py --cfg cfgs/gcd_offset.yaml --log_dir </path/to/logs> --tag CenterLPS_KITTI_offset

For nuScenes:

$ python cfg_train.py --cfg cfgs/nuscenes/gcd_offset.yaml --log_dir </path/to/logs> --tag CenterLPS_nuScenes_offset

The training scripts could be found in ./scripts/*_train.sh. Before using the training scripts, please use the pretrained model (of step 2) in the folder weights or use the model trained (in step 2). For training with a single GPU, you can also use the following command. For SemanticKITTI:

$ python cfg_train.py --cfg cfgs/gcd_offset.yaml --log_dir </path/to/logs> --tag CenterLPS_KITTI --pretrained_ckpt weights/kitti_backbone_v2_offset.pth

For nuScenes:

$ python cfg_train.py --cfg cfgs/nuscenes/gcd_offset.yaml --log_dir </path/to/logs> --tag CenterLPS_nuScenes --pretrained_ckpt weights/nuscenes_backbone_offset.pth

Distributed under the MIT License. See LICENSE for more information.

In our implementation, we refer to the following open-source databases:

• spconv
• DSNet
• GASN