An independent reproduction of LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation (ICML 2026 submission #1055, OpenReview P6fWeIVbwb).

Every dataset claim is verified by counting the released metadata directly. Every model claim is verified by running the authors' own released 9B checkpoints through the authors' own evaluation code on a real A100 — not taken on faith, and not re-implemented.

Total cost: ~$13 and ~7 GPU-hours on 1× A100-80GB.

Headline: the core of the paper reproduces. All four dataset-composition claims verify exactly from the release, the headline perceptual-quality metric reproduces to within 0.0002 SRCC, and zero-shot cross-dataset transfer reproduces near-exactly. One number does not reproduce — see below.

Text–video correspondence falls ~0.10 SRCC short of Table 2, and the gap is real.

The obvious explanation is small-sample noise, so I spent GPU budget specifically to test it: tripling the sample to n=4,500 moved the number down, not up. That rules out subset variance.

The pipeline itself is sound — the identical harness reproduces perceptual quality to ΔSRCC 0.0002, and the same LOVE-Corrrespondence checkpoint reproduces the paper's zero-shot T2VQA-DB number to within 0.005. So the checkpoint predicts correspondence sensibly; it just doesn't hit 0.7466 on AIGVE-60K. The most likely explanation is checkpoint identity — the released weights are not the paper's final Table-2 model (e.g. an earlier or "fast" variant), or Table 2 reflects a training/eval detail not captured in the public release.

The same caveat applies more mildly to QA: the exact final LOVE-QA checkpoint isn't published, so the closest released weight (LMM4AIGV-QA) was used, landing 5 points low.

Full-resolution PDF (1.52 m × 0.91 m).

📖 Read the full writeup as an interactive logbook →

The logbook is the chronological record — each claim's cells appear in the order the evidence arrived, including the hypotheses that later got refuted (e.g. the "subset variance" explanation for the correspondence gap, and the run that killed it). The docs/ pages below are the reconciled version: same evidence, stated as final conclusions.

git clone https://github.com/ACNoonan/love-repro-aigve60k
cd love-repro-aigve60k
pip install -r requirements.txt

bash scripts/fetch_data.sh          # ~50 MB of released metadata from the Hub
python scripts/analyze_dataset.py   # Claims 1, 2, 3, 6
python scripts/recompute_claim4.py  # Claim 4 Stage-1, from released predictions

Run from the repo root — the scripts resolve data_meta/, ref_metrics/ and outputs/ relative to the working directory.

Writes outputs/dataset_composition.json, outputs/claim6_taxonomy.json, and outputs/claim4_recompute.json. Committed copies of all three are already in outputs/ for comparison.

Each run is a self-contained HF Job on 1× A100-80GB. bootstrap*.sh installs pinned deps and decodes the driver into the container; the driver clones the upstream LOVE repo, patches it to build (see below), pulls the checkpoint and test videos, and runs the authors' eval.

# Claim 4 — perceptual quality (ASPECT=mos2 for correspondence, N_SUBSET=0 for all 9,000)
hf jobs run --flavor a100-large --secrets HF_TOKEN --timeout 2h -d \
  -e ASPECT=mos1 -e N_SUBSET=3000 \
  pytorch/pytorch:2.4.1-cuda12.1-cudnn9-devel bash -c "$(cat scripts/bootstrap.sh)"

# Claim 4 — QA accuracy
hf jobs run --flavor a100-large --secrets HF_TOKEN --timeout 2h -d \
  -e N_SUBSET=1500 \
  pytorch/pytorch:2.4.1-cuda12.1-cudnn9-devel bash -c "$(cat scripts/bootstrap_qa.sh)"

# Claim 5 — zero-shot on T2VQA-DB
hf jobs run --flavor a100-large --secrets HF_TOKEN --timeout 2h -d \
  -e BENCH=t2vqa -e N_SUBSET=1500 \
  pytorch/pytorch:2.4.1-cuda12.1-cudnn9-devel bash -c "$(cat scripts/bootstrap_zs.sh)"

HF_TOKEN is read from the environment and needs write access if you set OUT_REPO to your own dataset repo (it defaults to mine).

The drivers apply these automatically — recorded here because they're the practical blockers to reproducing this work:

• pytorchvideo is imported but unused → stubbed, avoiding an unnecessary dependency.
• temporal.pth is not shipped with the released checkpoints → that init is skipped.
• patch/__init__ pulls in flash-attn → trimmed, avoiding a long source build.

scripts/     reproduction harness — 3 GPU job drivers + bootstraps, 2 local analyses
outputs/     all result JSONs (local + every GPU run, including smoke runs)
docs/        per-claim evidence pages
reference/   the authors' released Stage-1 metrics, used for cross-checking
poster/      conference poster (PNG, PDF, HTML source)

Raw per-video CSVs, subset manifests, and metrics for every GPU run are mirrored to adamcnoonan/love-repro-aigve60k on the Hub, keyed by the job IDs recorded in each outputs/gpu/*.json.

Not attempted: re-collecting or re-annotating the dataset, retraining LOVE, and re-scoring the 23 LMMs / 24 VQA metrics of the V2T leaderboard.

All models, data, and evaluation code are the work of the original authors (IntMeGroup). This repository contains only reproduction harness code and measurements. The AIGVE-60K dataset and LOVE checkpoints are governed by their own licenses on the Hub.

@article{love2025,
  title  = {LOVE: Benchmarking and Evaluating Text-to-Video Generation
            and Video-to-Text Interpretation},
  journal = {arXiv preprint arXiv:2505.12098},
  year   = {2025},
  url    = {https://arxiv.org/abs/2505.12098}
}

Reproduction by @ACNoonan. MIT licensed — see LICENSE.