Haoyuan Li, Qi Hu, Binjia Zhou, You Yao, Jiacheng Lin, Kailun Yang*, Peng Chen*
*Corresponding Authors
In this work, we propose the Cross-modality Fusion Mamba with Weather-removal (CFMW) to augment stability and cost-effectiveness under adverse weather conditions. Leveraging the proposed Perturbation-Adaptive Diffusion Model (PADM) and Cross-modality Fusion Mamba (CFM) modules, CFMW is able to reconstruct visual features affected by adverse weather, enriching the representation of image details. To bridge the gap in relevant datasets, we construct a new Severe Weather Visible-Infrared (SWVI) dataset, encompassing diverse adverse weather scenarios such as rain, haze, and snow.
- Download the four compressed files here. They require about 20GB of storage space.
- Run the following command to merge the two files into one and uncompress it. This will produce a folder named
SWVIcontaining 60K visible images, infrared images and visible images with weather-influence.
cat SWVI.* > SWVI.zip
unzip SWVI.zipSWVI/
├── README.md
└── infrared
├── 03205.png
├── 03206.png
├── 03207.png
└── ...
└── visible
├── 03205.png
├── 03206.png
├── 03207.png
└── ...
└── visible_with_weather
├── 03205.png
├── 03206.png
├── 03207.png
└── ...
└── labels
├── 03205.txt
├── 03206.txt
├── 03207.txt
└── ...
├── train.txt
└── val.txt
conda create -n your_env_name python=3.8 -y
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
pip install -e causal_conv1d>=1.1.0
pip install -e mamba-1p1p1
cd PADM/
export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}
python train_diffusion.py
cd PADM/
python eval_diffusion.py
export CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}
python train.py
python detect_twostram.py
python test.py
If you find our work and this codebase helpful, please consider starring this repo 🌟 and cite:
@ARTICLE{11077409,
author={Li, Haoyuan and Hu, Qi and Zhou, Binjia and Yao, You and Lin, Jiacheng and Yang, Kailun and Chen, Peng},
journal={IEEE Transactions on Circuits and Systems for Video Technology},
title={CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather},
year={2025},
doi={10.1109/TCSVT.2025.3587918}}