This repository contains the implementation of the 3MT-RoadSeg method in Pytorch. 3MT-RoadSeg is a fast and accurate method that does not need any preprocessing and uses only raw sensor inputs.
The training setup in the repo is now available for the KITTI and Cityscapes datasets. Training setups will be created for new data sets in the coming period.
The package versions used during code development are as follows:
- Python: 3.7
- Pytorch: 1.9.0
- Cudatoolkit: 10.2
- Torchvision: 0.10.0
- Opencv-python: 4.5.3.56
You can download the KITTI road dataset from KITTI and the Cityscapes road dataset from Cityscapes. Depth images can be obtained from the SNE-RoadSeg. You can access the 3-channel ADIs from here.
3MT-RoadSeg
|-- data
| |-- to
| | |-- databases
| | | |-- KITTIRoad
| | | | |-- training
| | | | | |-- ...
| | | | | -- testing
| | | | | |-- ...
| | | |-- CityScapes
| | | | |-- training
| | | | | |-- ...
| | | | |-- testing
| | | | | |-- ...
...
Configuration files are located in the configs/ directory. You have the option to alter the training settings from this directory if you prefer. You can split the original training set into a new training set and a validation set as you wish. Then run the script below:
train.py --config_env configs/env.yml --config_exp configs/$DATASET/$BACKBONE/$MODEL.yml
You can perform a test on the validation set created by splitting the training set. You will need the chechpoint.pth.rar file for this.
test.py ---config_env configs/env.yml --config_exp configs/$DATASET/$BACKBONE/$MODEL.yml
Additionally, if you only want to output for single input, you can do this using test_singleInput.py like this:
test_singleInput.py ---config_env configs/env.yml --config_exp configs/$DATASET/$BACKBONE/$MODEL.yml
You can use the same split we used for the validation set generated from the training set. The train & test split is as follows:
| Train Split | | Test Split |
|------------------------------------------------| |------------------------------------------------|
um_000000.png | umm_000000.png | uu_000000.png um_000068.png | umm_000068.png | uu_000068.png
... | ... | ... ... | ... | ...
... | ... | ... ... | ... | ...
um_000067.png | umm_000067.png | uu_000067.png um_000094.png | umm_000095.png | uu_000097.png
The following datasets and tasks are supported.
| Dataset | Segmentation | Depth | Normals |
|---|---|---|---|
| KITTI | Y | Y | Aux |
| Cityscapes | Y | Aux | Y |
The following models are supported.
| Backbone | HRNet-w18 | HRNet-w32 | HRNet-w48 |
|---|---|---|---|
| 3MT-RoadSeg | Y | Y | Y |
You can download the weights belonging to the pre-trained HRNet backbones from source.
This repository has been built upon the foundation of repository Multi-Task-Learning-PyTorch.
@ARTICLE{10182336,
author={Milli, Erkan and Erkent, Özgür and Yılmaz, Asım Egemen},
journal={IEEE Robotics and Automation Letters},
title={Multi-Modal Multi-Task (3MT) Road Segmentation},
year={2023},
volume={8},
number={9},
pages={5408-5415},
doi={10.1109/LRA.2023.3295254}}