FG-BMK

📄 This repository contains the data and evaluation code for FG-BMK. For an interactive leaderboard, dataset overview, and key findings, visit the project page.

• 🔥 (new) An extended version of our paper is now on arXiv.
• ✨ Major benchmark update — we added new dataset and expanded the suite with additional experimental validation and analysis.
• 🎉 Our paper has been accepted to ICLR 2026! The ICLR version is available on arXiv.
• 🚀 We released our FG-BMK benchmark!

FG-BMK is a comprehensive fine-grained evaluation benchmark — 1.01 million questions over 0.28 million images — providing a robust test bed for evaluating LVLMs. It incorporates two complementary evaluation paradigms:

• 🧑 Human-oriented evaluation — employs dialogue-like interactions to assess a model's ability to understand and respond to fine-grained visual queries in a conversational context.
• 🤖 Machine-oriented evaluation — focuses on two core fine-grained vision tasks, image retrieval and recognition, to directly measure the feature-representation capabilities of LVLMs.

Compared with existing efforts — which primarily focus on fine-grained classification or use a limited number of questions — FG-BMK enables a comprehensive assessment of LVLMs' fine-grained feature representation and semantic recognition abilities. Our evaluations of eight open-source LVLMs/VLMs uncover key findings on the influence of training paradigms, modality alignment, perturbation susceptibility, and fine-grained category reasoning on task performance.

Before running inference, download the corresponding dataset images. Links to the source dataset projects are provided below — you can download the datasets to any location:

• Human-oriented evaluations — questions are pre-generated for each image, e.g. benchmark/human-oriented/attribute_recognition/cub_attribute_questions.jsonl.
• Machine-oriented evaluations — dataset categories and the corresponding train.csv / test.csv splits live in directories such as benchmark/machine-oriented/aircraft.

Our evaluation covers eight LVLMs/VLMs. The table below lists the project and checkpoint for each model, to help you configure the environment and download the relevant weights:

To use your own model and produce final answers, first modify the model-loading code in human_evaluation_demo.py to adapt it to your model. Here is an example of loading the InternVL model:

# Load InternVL model, tokenizer, and image processor
from transformers import AutoModel, AutoTokenizer, CLIPImageProcessor
model = AutoModel.from_pretrained(
    args.model_path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True, use_fast=False)
image_processor = CLIPImageProcessor.from_pretrained(args.model_path)

The model then answers the questions through its inference code, e.g.:

# Load images
image = Image.open(image_path).resize((448, 448))
pixel_values = image_processor(images=image, return_tensors='pt').pixel_values.to(torch.bfloat16).cuda()
# Generate response
generation_config = dict(max_new_tokens=1024, do_sample=True)
response = model.chat(tokenizer, pixel_values, prompt_text, generation_config)

After modifying the loading code, configure model-path (checkpoint), question-file, image-folder (path to the dataset), and answers-file (output path) in run_human_demo.sh, then run:

bash run_human_demo.sh
# The code splits the question file by the number of GPUs and runs inference concurrently.

The outputs are merged into a single file in the following format:

{"question_id": 1, "image": "images/001.Black_footed_Albatross/Black_Footed_Albatross_0078_796126.jpg", "prompt": "Is the genus of the object geococcyx? Answer with yes or no.", "text": "No", "class": "no", "category": "generic"}
{"question_id": 2, "image": "images/001.Black_footed_Albatross/Black_Footed_Albatross_0003_796136.jpg", "prompt": "Is the genus of the object raven? Answer with yes or no.", "text": "No", "class": "no", "category": "generic"}

Finally, compute accuracy with answer_acc.py:

python answer_acc.py

See the example output for a detailed prediction-file format. We also provide reference inference code for Qwen-VL and BLIP-2.

To evaluate an LVLM's feature-representation ability, first modify the feature-extraction code in models.py. Here is an example of CoCa feature extraction:

def coca(model_name, pretrained, cache_dir):
    from open_clip import create_model_and_transforms

    def _hook(self, _, input, output):
        self.feat.append(output.transpose(0, 1))

    def get_intermediate_layers(self, x, n=1, return_class_token=True):
        self.feat = []
        self(x)
        class_tokens = [out[:, 0] for out in self.feat]
        outputs = [out[:, 1:] for out in self.feat]
        return tuple(zip(outputs, class_tokens))

    model, _, preprocess = create_model_and_transforms(model_name, pretrained, cache_dir=cache_dir)
    model = model.visual
    model.eval()
    model.cuda()
    model.__class__._hook = _hook
    model.__class__.get_intermediate_layers = get_intermediate_layers
    model.transformer.resblocks[-2].register_forward_hook(model._hook)
    model.transformer.resblocks[-1].register_forward_hook(model._hook)
    return model

In this module we use _hook and the defined get_intermediate_layers method to extract visual features from the last two layers of the vision encoder. We then concatenate the cls token and image tokens in the predefined order and return an instance of the CoCa model. Examples of visual feature extraction using EVA-CLIP and Qwen-VL are already provided in model.py.

Once modified, import the model in eval_linear.py or eval_retrieval.py:

from dinov2.utils.config import setup
from models import coca
torch.backends.cudnn.benchmark = True
model = coca('coca_ViT-L-14', 'laion2b_s13b_b90k', '.cache')
config = setup(args)
autocast_dtype = torch.float16

Then run the demo:

python eval_linear.py
# or
python eval_retrieval.py

The output log will look like:

I20240504 13:34:23 16157 dinov2 helpers.py:103] Training  [    0/10000]  eta: 8:13:21  loss: 143.1147 (143.1147)  lr: 0.0005 (0.0005)  time: 2.960182  data: 2.449736  max mem: 2711
...

After the training process completes, the code automatically runs inference on the test set of the fine-grained dataset and reports the results.

If you find FG-BMK useful for your research, please consider citing:

@article{yu2026fgbmk,
  title   = {Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis},
  author  = {Yu, Hong-Tao and Xie, Chen-Wei and Peng, Yuxin and Belongie, Serge and Wei, Xiu-Shen},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
  year    = {2026}
}