This repository contains the code for running our FIRe model presented in our ICLR'22 paper:
@inproceedings{superfeatures,
title={{Learning Super-Features for Image Retrieval}},
author={{Weinzaepfel, Philippe and Lucas, Thomas and Larlus, Diane and Kalantidis, Yannis}},
booktitle={{ICLR}},
year={2022}
}The code is distributed under the CC BY-NC-SA 4.0 License. See LICENSE for more information. It is based on code from HOW, cirtorch and ASMK that are released under their own license, the MIT license.
After cloning this repository, you must also have HOW, cirtorch and ASMK and have them in your PYTHONPATH.
- install HOW
git clone https://github.com/gtolias/how
export PYTHONPATH=${PYTHONPATH}:$(realpath how)
- install cirtorch
wget "https://github.com/filipradenovic/cnnimageretrieval-pytorch/archive/v1.2.zip"
unzip v1.2.zip
rm v1.2.zip
export PYTHONPATH=${PYTHONPATH}:$(realpath cnnimageretrieval-pytorch-1.2)
- install ASMK
git clone https://github.com/jenicek/asmk.git
pip3 install pyaml numpy faiss-gpu
cd asmk
python3 setup.py build_ext --inplace
rm -r build
cd ..
export PYTHONPATH=${PYTHONPATH}:$(realpath asmk)
- install dependencies by running:
pip3 install -r how/requirements.txt
- data/experiments folders
All data will be stored under a folder fire_data that will be created when running the code; similarly, results and models from all experiments will be stored under folder fire_experiments
To evaluate on ROxford/RParis our model trained on SfM-120k, simply run
python evaluate.py eval_fire.yml
With the released model and the parameters found in eval_fire.yml, we obtain 90.3 on the validation set, 82.6 and 62.2 on ROxford medium and hard respectively, 85.2 and 70.0 on RParis medium and hard respectively.
Simply run
python train.py train_fire.yml -e train_fire
All training outputs will be saved to fire_experiments/train_fire.
To evaluate the trained model that was saved in fire_experiments/train_fire, simply run:
python evaluate.py eval_fire.yml -e train_fire -ml train_fire
For reproducibility, we provide the following model weights for the architecture we use in the paper (ResNet50 without the last block + LIT):
- Model pre-trained on ImageNet-1K (with Cross-Entropy, the pre-trained model we use for training FIRe) (link)
- Model trained on SfM-120k trained with FIRe (link)
They will be automatically downloaded when running the training / testing script.
For convenience, we provide a dockerfile. You can build it with
docker build --tag naver/fire .It does not contain the fire_data nor fire_experiments so these need to be stored outside.
In evaluate.py, the options --data-folder and --exp-folder can be used to overwrite these paths.
example:
docker run --gpus all --rm -it --ipc=host --mount type=bind,source=/local/fire,target=/local/fire --entrypoint bash naver/fire
python evaluate.py eval_fire.yml --data-folder /local/fire/fire_data --exp-folder /local/fire/fire_experimentsWith kapture_compute_pairs.py you can compute pairs from datasets that are provided in kapture format (link to kapture github) using FIRe or HOW. These pairs can be used to, e.g., run the kapture visual localization pipeline (link to kapture-localization github).
--codebook-cache-path can be used to cache the codebook. It only needs to be computed once per model.
--ivf-cache-path can be used to cache the ivf database. It needs to be computed once per model per dataset (mapping images).
--model-load, --data-folder can be used to overwrite demo_eval.net_path and demo_eval.fire_data. Note that demo_eval.exp_folder and evaluation.local_descriptor.datasets are ignored.
example: extracting top50 FIRe pairs, and top50 HOW pairs for GangnamStation_B2
docker run --gpus all --rm -it --ipc=host --mount type=bind,source=/local/fire,target=/local/fire --entrypoint bash naver/fire
# prepare dataset
mkdir /local/fire/kapture_datasets
cd /local/fire/kapture_datasets
kapture_download_dataset.py update
kapture_download_dataset.py install "GangnamStation_B2*"
# read license terms and type y [enter] to agree
cd GangnamStation/B2/release
kapture_merge.py -v info \
-i test validation \
-o query_all \
--image_transfer link_relative
# extract FIRe pairs
cd /opt/src/fire
# map -> map pairs
python3 kapture_compute_pairs.py -v debug \
--parameters eval_fire.yml \
--model fire \
--data-folder /local/fire/fire_data \
--codebook-cache-path /local/fire/fire_codebook \
--ivf-cache-path /local/fire/kapture_datasets/GangnamStation/B2/release/fire_ivf \
--map /local/fire/kapture_datasets/GangnamStation/B2/release/mapping/ \
-o /local/fire/kapture_datasets/GangnamStation/B2/release/pairsfile/mapping/fire_top50.txt \
--topk 50
# query -> map pairs
python3 kapture_compute_pairs.py -v debug \
--parameters eval_fire.yml \
--model fire \
--data-folder /local/fire/fire_data \
--codebook-cache-path /local/fire/fire_codebook \
--ivf-cache-path /local/fire/kapture_datasets/GangnamStation/B2/release/fire_ivf \
--map /local/fire/kapture_datasets/GangnamStation/B2/release/mapping/ \
--query /local/fire/kapture_datasets/GangnamStation/B2/release/query_all \
-o /local/fire/kapture_datasets/GangnamStation/B2/release/pairsfile/query/fire_top50.txt \
--topk 50
# extract HOW pairs
cd /opt/src/fire
# you can use the same data-folder as for fire
# map -> map pairs
python3 kapture_compute_pairs.py -v debug \
--parameters ../how/examples/params/eccv20/eval_how_r50-_1000.yml \
--model how \
--data-folder /local/fire/fire_data \
--codebook-cache-path /local/fire/how_codebook \
--ivf-cache-path /local/fire/kapture_datasets/GangnamStation/B2/release/how_ivf \
--map /local/fire/kapture_datasets/GangnamStation/B2/release/mapping/ \
-o /local/fire/kapture_datasets/GangnamStation/B2/release/pairsfile/mapping/how_top50.txt \
--topk 50
# query -> map pairs
python3 kapture_compute_pairs.py -v debug \
--parameters ../how/examples/params/eccv20/eval_how_r50-_1000.yml --model fire \
--data-folder /local/fire/fire_data \
--codebook-cache-path /local/fire/how_codebook \
--ivf-cache-path /local/fire/kapture_datasets/GangnamStation/B2/release/how_ivf \
--map /local/fire/kapture_datasets/GangnamStation/B2/release/mapping/ \
--query /local/fire/kapture_datasets/GangnamStation/B2/release/query_all \
-o /local/fire/kapture_datasets/GangnamStation/B2/release/pairsfile/query/how_top50.txt \
--topk 50