MIRAGE and Logos

MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM

paper data Logos_train_data Logos-3B Logos-7B

Our evaluation code is based on VIC

1. Download eval data into eval code directory, then switch to this directory

2. execute following code for inference

python -m Vic.benchmark_test -p mirage.tsv -i original (for reasoning mllms)

or 

python -m Vic.benchmark_test -p mirage.tsv -i cot (for vanilla mllms)

3. Evaluation, specifically, for accuracy:

python -m Vic.benchmark_eval -b mirage -p output_inference_results.tsv

We implement our train code based on OpenRLHF. We will upload our version soon.

The training data has been released in Logos_train_data.