💡 How to evaluate Text-to-Video Generation Models properly?

Download with CLI:

huggingface-cli download anonymousdb/AIGVE-60K --repo-type dataset --local-dir ./AIGVE-60K

This leaderboard presents the performance of 30 models on the AIGVE-60K benchmark, evaluating across three key dimensions:

• 🎨 Perceptual Quality
• 🔄 Text-to-Video Correspondence
• ❓ Task-specific Accuracy

Final Overall Rank is computed by summing the individual ranks across these three dimensions. The top 3 models are marked with 🥇🥈🥉. All model names are bolded for clarity.

Clone the repository:

git clone https://github.com/IntMeGroup/LOVE.git

Create and activate a conda environment:

conda create -n LOVE python=3.9 -y
conda activate LOVE

Install dependencies:

pip install -r requirements.txt

Install flash-attn==2.3.6 (pre-built):

pip install flash-attn==2.3.6 --no-build-isolation

Or compile from source:

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
git checkout v2.3.6
python setup.py install

huggingface-cli download anonymousdb/AIGVE-60K data.zip --repo-type dataset --local-dir ./
unzip data.zip -d ./data

huggingface-cli download OpenGVLab/InternVL3-9B --local_dir OpenGVLab/InternVL3-9B
huggingface-cli download anonymousdb/LOVE-pretrain temporal.pth ./

sh shell/st1_train.sh

sh shell/st2_train.sh

sh shell/train_qa.sh

huggingface-cli download anonymousdb/LOVE-Perception --local-dir ./weights/stage2/stage2_mos1
huggingface-cli download anonymousdb/LOVE-Correspondence --local-dir ./weights/stage2/stage2_mos2
huggingface-cli download anonymousdb/LOVE-QA --local-dir ./weights/qa

sh shell/eval_score.sh

sh shell/eval_qa.sh

huggingface-cli download anonymousdb/LOVE-Perception --local-dir ./weights/stage2/stage2_mos1
huggingface-cli download anonymousdb/LOVE-Correspondence --local-dir ./weights/stage2/stage2_mos2

1. Refine the /data/infer_perception.json file with the correct path:

"root": your_path_to_videos

or infer selected videos in video_names.txt 2. Refine the /data/data/infer_perception2.json file with the correct path:

"root": your_path_to_videos
"video_name_txt": video_names.txt

and change the shell/infer_perception.sh line30 to data/infer_perception2.json 3. Refine the /data/infer_correspondcence.json file with the correct path:

"root": your_path_to_videos
"video_name_txt": "video_names.txt",
"prompt_txt": "prompt.txt",

Refine the shell/infer_perception.sh line27 to your_download_model_pretrained_weight_path

sh shell/infer_perception.sh

Refine the shell/infer_correspondence.sh line27 to your_download_model_pretrained_weight_path

sh shell/infer_correspondence.sh

This leaderboard presents the performance of 48 models on the AIGVE-60K benchmark, evaluating across three key dimensions:

• 🎨 Perception SRCC: Spearman correlation of perceptual quality.
• 🔄 Correspondence SRCC: Spearman correlation of text-to-video alignment.
• ❓ QA Accuracy: Accuracy of question answering.

Final Overall Rank is computed by summing the individual ranks across these three dimensions. The top 3 models are marked with 🥇🥈🥉. All model names are bolded for clarity.

This section lists 30 representative T2V generation models, including both commercial close-source models and open-source lab models, with links to their official or GitHub pages.

This repository provides a comprehensive list of ** Vision-to-Text (V2T) interpretation models**, covering conventional video quality assessment models, learning-based image-text alignment models, large multimodal models (LMMs), and proprietary foundation models. Each method is annotated with its category and accompanied by a corresponding GitHub or official URL.

• Conventional VQA Metrics
• ♣️ Classical VQA Models
• ❤️ Learning-based Scoring Models
• ⭐ Large Multimodal Models (LMMs)
• 🔺 Proprietary Foundation Models (Closed-source)

run:

mos15.m

For BMPRI, BPRI, BRISQUE, HOSA, NIQE, QAC run:

videobench.m

The current rankings are based on data we obtained from randomly-selected professional annotators, and we do not intend to offend the developers of these excellent T2V and V2T models. Although our model shows promising scalability in evaluating AIGVs generated by new prompts and previously unseen T2V models, the effectiveness in real-world applications remains an open question.

We hope that our benchmark and dataset will contribute to the advancement of:

• 🎨 T2V Generation
• 📊 T2V Evaluation
• 🔁 V2T Interpretation

Thanks to the original authors of all the models listed here. This is a curated list intended to help researchers and developers in the T2V generation,V2T interpretation and multimodal quality assessment communities.

• ✅ Release the training code
• ✅ Release the evaluation code
• ✅ Release the AIGVE-60K Database