Arxiv | Kaggle |

This is the official repository for our paper "AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection" (arxiv paper link).

• [2025.02] Our paper has been accepted by the IEEE Transactions on Multimedia (TMM)!

The paper mentions that the data for the training and test sets of the SMOD Dataset are incorrect, and that all of our experiments are based on the existing versions on Kaggle. The dataset we used has 2676 samples in the test set and 6000 samples in the training set. we are sorry for the mistake.

1. We use the MMDetection toolbox to detect pedestrians on the KAIST, LLVIP and SMOD dataset. Please follow the MMDetection documents to install environments.

   In our environment, we use:

   python==3.9.1
   pytorch==1.12.1+cu116+cudnn8_0
   torchvision==0.13.1+cu116
   mmcv==2.0.1
   mmdet==3.1.0
   mmengine==0.8.4
   

2. Go to the folder where mmdetection is located and clone our project to the projects folder in that directory.

   cd mmdetection
   cd projects
   git clone https://github.com/bigD233/AMFD.git
   

All Datasets and Models we provided are upload in google cloud.

• KAIST: KAIST dataset has been updated by several previous works. We upload this dataset and improved annotations for your convenience in using our code. cloud link
• LLVIP: The origin data you can download in its from its official repository. The coco-format annotation you can download from cloud link.
• SMOD: A new multispectral object detection dataset propsed by us. You can download from Kaggle

We provide the teacher checkpoint "Teacher_Fasterrcnn_7_66.pth" and the studnet checkpoint "single_fasterrcnn_7_23.pth" trained on KAIST dataset. cloud link

Teacher_Fasterrcnn_7_66.pth ---> Trained by Teacher_Fasterrcnn_r50_fpn_1x_kaist_thermal_first.py
single_fasterrcnn_7_23.pth  ---> Trained by Student_Fasterrcnn_r18_fpn_1x_kaist_amfd.py

We take the example of inference on the KAIST dataset:

1. Download checkpoints you need for test. You can refer to Datasets and Models. Here we need the checkpoint "resnet18-5c106cde.pth" and "single_fasterrcnn_7_23.pth" to initialize the weights.

2. Modify the corresponding paths in the configuration file, including the file save path, the path of the checkpoint used for initialization, and the dataset path.

3. Inference.

   cd mmdetection
   python ./tools/test.py ./projects/AMFD/config/KAIST/Student_Fasterrcnn_r18_fpn_1x_kaist.py {your_path}/single_fasterrcnn_7_23.pth
   

1. Download the checkpoint of teacher network. You can refer to Datasets and Models. Here we need the checkpoint "resnet18-5c106cde.pth" (for the student) and "Teacher_Fasterrcnn_7_66.pth" (as the teacher checkpoint) to initialize the weights. If you want to train a teacher network by yourself:

   python ./tools/train.py ./projects/AMFD/config/KAIST/Teacher_Fasterrcnn_r50_fpn_1x_kaist_thermal_first.py
   

2. Modify paths in the config "Student_Fasterrcnn_r18_fpn_1x_kaist_amfd.py"

3. Distill

   python ./tools/train.py ./projects/AMFD/config/KAIST/Student_Fasterrcnn_r18_fpn_1x_kaist_amfd.py
   

If you find our AMFD useful, please cite our paper:

  @article{amfd,
  	title={AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection},
  	author={Chen, Zizhao and Qian, Yeqiang and Yang, xiaoxiao and Wang, Chunxiang and Yang, Ming},
  	journal={arXiv preprint arXiv:2405.12944},
  	year={2024}}

email: czz-000@sjtu.edu.cn, qianyeqiang@sjtu.edu.cn, xiaoxiaoyang7021@gmail.com