MUSTER

MUSTER: A Multi-scale Transformer-based Decoder for Semantic Segmentation

The core codes can be found at:

MUSTER/mmseg/models/decode_heads/MUSTER_head.py

We use MMSegmentation v0.29.0 as the codebase.

For install , please refer to the guidelines in MMSegmentation v0.29.0.

An example (works for me): CUDA 11.6 and pytorch 1.12.1

Step 0. Install MMCV using MIM.

pip install -U openmim
mim install mmcv-full

Step 1. Install MUSTER.

# The workdir of the terminal should be MUSTER/
pip install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.

For dataset preparation, please refer to the guidelines in this link.

It is recommended to symlink the dataset root to MUSTER/data. If your folder structure is different, you may need to change the corresponding paths in config files.

The fold structure is recommended to be:

MUSTER
├── data
│   ├── cityscapes
│   │   ├── leftImg8bit
│   │   │   ├── train
│   │   │   ├── val
│   │   ├── gtFine
│   │   │   ├── train
│   │   │   ├── val
│   ├── ade
│   │   ├── ADEChallengeData2016
│   │   │   ├── annotations
│   │   │   │   ├── training
│   │   │   │   ├── validation
│   │   │   ├── images
│   │   │   │   ├── training
│   │   │   │   ├── validation

The data could be found here after registration.

By convention, **labelTrainIds.png are used for cityscapes training. MMsegmentation provided a script based on cityscapesscripts to generate **labelTrainIds.png.

# --nproc means 8 process for conversion, which could be omitted as well.
python tools/convert_datasets/cityscapes.py data/cityscapes --nproc 8

The training and validation set of ADE20K could be download from this link. You may also download test set from here.

Because of anonymity, we cannot give trained weights at present.

Example: evaluate MUSTER with Swin Transformer on ADE20K:

# Single-gpu testing
python tools/test.py configs/MUSTER/MUSTER_swin-base_ade20k_160k.py /path/to/checkpoint_file --show

Example: evaluate MUSTER with Swin Transformer on Cityscapes:

# Single-gpu testing
python tools/test.py configs/MUSTER/MUSTER_swin-base_ade20k_160k.py /path/to/checkpoint_file --show

Example: train MUSTER with Swin Transformer on ADE20K:

# Single-gpu training
python tools/train.py configs/MUSTER/MUSTER_swin-base_ade20k_160k.py

Please check the LICENSE file.

Please check CITATION.cff.