💡 Interested in evaluating your model on ReasonMap or ReasonMap-Plus?
📩 Contact us fscnkucs@gmail.com
🙋 Please let us know if you find out a mistake or have any suggestions!
🌟 If you find this resource helpful, please consider to star this repository and cite our research!
- 2025-09-30: 🚀 We released ReasonMap-Plus for the following research - RewardMap!
- 2025-05-15: 🚀 We released evaluation code and our website online!
- 2025-05-15: 🚀 We released ReasonMap!
If you face any issues with the installation, please feel free to open an issue. We will try our best to help you.
conda env create -f reasonmap-py310.yamlYou can download ReasonMap and ReasonMap-Plus from HuggingFace.
You can evaluate the model performance on ReasonMap by running the following command:
## ReasonMap Evaluation
# open-source models
bash script/run.sh
# closed-source models
bash script/run-closed-models.sh
## ReasonMap-Plus Evaluation
bash script/run_plus.sh
# after running the above scripts, you can analyze the results by:
python cal_metrics.pyIf you find this benchmark useful in your research, please consider citing our paper:
@article{feng2025can,
title={Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps},
author={Feng, Sicheng and Wang, Song and Ouyang, Shuyi and Kong, Lingdong and Song, Zikai and Zhu, Jianke and Wang, Huan and Wang, Xinchao},
journal={arXiv preprint arXiv:2505.18675},
year={2025}
}
# further research
@article{feng2025rewardmap,
title={RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning},
author={Feng, Sicheng and Tuo, Kaiwen and Wang, Song and Kong, Lingdong and Zhu, Jianke and Wang, Huan},
journal={arXiv preprint arXiv:2510.02240},
year={2025}
}