[📖 Paper] [🤗 V-Retrver-7B-model] [🤗V-Retrver-RFT-model] [🤗V-Retrver-SFT-model] [🤗 V-Retrver-train-data] [🤗 V-Retrver-eval-data]
We introduce V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.
To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective.
Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.
All code, models, and data are fully released.
- [2026/9/21] V-Retrver was accepted by EMNLP 2026 Main !!!
- [2026/2/06] We release the code, model, data of V-Retrver
- Support Qwen3-VL/Qwen2.5-VL Training
- Provide full pipeline (dataset, SFT training, RFT training, RL training, evaluation, etc)
V-Retrver-7B demonstrates strong performance across multiple multimodal retrieval benchmarks.
Some reasoning examples are as follows.
cd verltool
git submodule update --init --recursive
conda create --name v-retrver python=3.10
conda activate v-retrver
pip install -e verl
pip install -e ".[vllm,acecoder,torl,search_tool]"
pip install "flash-attn==2.8.3" --no-build-isolation
We recommend to use the popular LLaMA-Factory to perform SFT on our cold-start data.
- Install LLaMA-Factory.
- Follow the instructions in LLaMA-Factory to configure the cold-start data in
data/dataset_info.json, as shown below, then copy the config filesftconfig/qwen2_5vl_retrv_full_sft.yamlinto your LLaMA-Factory codebase.
"V-Retrver_SFT": {
"file_name": "[YOUR_DATASET_FOLDER]/V-Retrver_SFT.json",
"formatting": "sharegpt",
"columns": {
"messages": "conversations",
"images": "images"
},
"tags": {
"role_tag": "from",
"content_tag": "value",
"user_tag": "human",
"assistant_tag": "gpt",
"system_tag": "system"
}
}
- Train Cold-start data with the training configs.
llamafactory-cli train sft_configs/qwen2_5vl_retrv_full_sft.yaml
In this stage, we improve reasoning reliability through Rejection Sampling.The training process and configurations for this stage are identical to Stage 1 (SFT). You simply need to prepare the RSFT dataset and follow the same training steps described in Stage 1.
The reinforcement learning is based on the RSFT model. You could either use the model produced in stage 1, or directly download it from V-Retrver/V-Retrver-RFT-7B.
cd verltool
bash examples/train/v-retrver/train_qwen25vl.sh
It should be able to run under 8 A800 GPUs with 80GB memory. From more details,please refer to verl-tool.
Tips:
- if output shared memory, try lower the
data.dataloader_num_workers - if out of cuda memory during vllm rollout, try set
actor_rollout_ref.rollout.enforce_eager=True, might be slower. - if out of cuda memory during training, try lower the
use_dynamic_bs=False.
We recommend using our provided json files and scripts for easier evaluation.
The json files can be downloaded at: [🤗 V-Retrver-eval-data].
You can conduct inference on all benchmarks using the following scripts
cd verltool
bash examples/train/AdaTooler-V/eval.sh
Use sliding_window_processor.py for reranking over a large candidate pool. This script splits the input Parquet file into specific windows (default strategy: 30-49, 20-39, etc.), calls the core eval.sh for each window, and merges the results to update the final rankings.
cd verltool
python sliding_window_processor.py \
--input_parquet /path/to/your/input_data.parquet \
--output_parquet /path/to/save/reranked_data.parquet \
--window_size 20
Use rerank_top20_processor.py (or the corresponding mode) to directly evaluate and rerank the top-20 candidates, skipping the sliding window logic. This script also wraps the core eval.sh to perform the evaluation.
cd verltool
python rerank_top20_processor.py \
--input_parquet /path/to/your/input_data.parquet \
--output_parquet /path/to/save/top20_reranked.parquet \
--mode top20
We sincerely appreciate the contributions of the open-source community. The related projects are as follows: verl-tool, verl, LLaMA-Factory, LamRA
If you find our work helpful for your research, please consider citing our work.
@article{chen2026v,
title={V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval},
author={Chen, Dongyang and Wang, Chaoyang and SU, Dezhao and Xiao, Xi and Zhang, Zeyu and Xiong, Jing and Li, Qing and Shang, Yuzhang and Kan, Shichao},
journal={arXiv preprint arXiv:2602.06034},
year={2026}
}