First, ensure you have stable connection to huggingface. The dataset can be found at this link. You can use the following code to download the dataset programmatically:

from datasets import load_dataset
dataset = load_dataset("Ruixiang/FineArtBench")

Download Link: 百度网盘 (Baidu NetDisk) | Google Drive

After downloading, please extract the dataset and put it under the data/ folder of the root directory. The directory structure should be as follows:

MLLM4Art/
├── data/
│   ├── 2AFC/               #the pair-wise comparison tasks
│   ├── base/               #the content, style images and their annotations
│   ├── human_annotation/   #the human aesthetic judgment for 2AFC tasks
│   ├── painting/           # the paintings generated by different models

Create a virtural environment and install the required packages:

pip install -r requirements.txt

To test your own local model on the 2AFC tasks of the FineArtBench, please refer to custom_model_evaluate.py. Specifically, you shall implement the Evaluator interface with your own model inference logic and map the model prediction to pairwise judgment. Then you can run the benchmark with the following command:

python custom_model_evaluate.py 

To test API-based MLLMs (e.g. ChatGPT), please refer to mllm_API_evaluate.py. You need to set up your own API key and base URL in the script. The exemplar config can be found in ./APIConfig/. Then you can run the benchmark with the following command:

python mllm_API_evaluate.py --config <REPLACE_WITH_YOUR_CONFIG_PATH>

The expected output is a JSON file in the same format as ./data/2AFC/2AFC_global_N_5000.json, but with the winner field filled according to your model's predictions. I suggest always evaluate all 5,000 pairwise tasks, as the correlation script will handle the missing human annotations.

Once you have the model predictions in the required JSON format, you can evaluate the correlation performance with human judgments using benchmark.py. This is the script that gives you the quantitative scores (correlation and statistical significance). You can run the benchmark with the following command:

python benchmark.py --human_annotation <PATH_TO_HUMAN_ANNOTATION_JSON> \
--model_annotation <PATH_TO_YOUR_MODEL_PREDICTION_JSON> --mode <global/instance>

The --human_annotation annotation can be found in ./data/human_annotation/ folder. The --mode argument specifies whether to compute global (per-artist) correlation or instance-level (per-task) correlation, which corresponds to the two columns in Table 1 of our paper. For the global correlation, please always report the score under 5 random splits.

For human annotation the json files without _paper suffix are recommended, which contains 40% more annotations and with additional quality control. For exactly reproducing the results as in our paper, please use the files with _paper suffix.