An expert-curated benchmark for vision-language models in veterinary pathology.

VIPER is the first expert-curated benchmark for evaluating vision-language models on rodent toxicologic pathology, a domain that anchors preclinical drug-safety assessment but has been almost entirely absent from the pathology-VLM evaluation ecosystem. The benchmark contains 1,251 questions over 419 H&E-stained rat histology images, spans 9 organs, and was authored and validated by board-certified veterinary pathologists (ECVP).

This repository is the official evaluation suite. A single command runs any vision-language model that speaks the OpenAI Chat Completions API (GPT, Claude, Gemini, vLLM, SGLang, llama.cpp, and friends) against the full benchmark and reproduces the paper's scoring exactly.

This project was developed by the Mahmood Lab at Harvard Medical School and Brigham and Women's Hospital, with veterinary pathology partners at COMPATH (University of Bern), UC Davis, the University of Augsburg, TU Dresden, UT MD Anderson Cancer Center, and the University of Lausanne. Funded in part by NIH NIGMS R35GM138216.

Weishaupt, de Brot, Asin, Grau-Roma, Reitsam, Song, Bang, Le, Kather, Mahmood, Jaume. VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology. 2026.

• Veterinary-pathology specific. 1,251 questions over 419 H&E rat ROIs from 9 organs, all authored by board-certified veterinary pathologists (ECVP).
• Visually grounded. Every question was anchored in visible morphology and adversarially filtered against image-free guessability.
• Three question formats. Multiple-choice (MCQ, 5 options × 5 cyclic rotations), KPrim (4 true/false statements), and free-text with rubric-guided LLM-judge scoring.
• Run anywhere. One CLI, one OpenAI-compatible endpoint, no PyTorch installation needed.
• Paper-aligned scoring. MCQ accuracy is the mean over 5 cyclic-shift permutations of the answer order; KPrim uses the ETH half-point rule; free-text uses the calibrated 0.7·accuracy + 0.3·completeness LLM judge.
• Reproducibility. Every results.json carries a SHA-256 fingerprint of the judge prompt, the package version, and the git hash of the eval suite.

git clone https://github.com/mahmoodlab/viper.git
cd viper
uv sync --all-extras

VIPER has zero GPU dependencies. Every model is reached over the OpenAI Chat Completions API. To evaluate an open-weight model, serve it with vLLM or SGLang and pass the local URL via --api-base.

export OPENAI_API_KEY=sk-...

# 30-second smoke test on 5 questions
viper-eval --model gpt-5.4-mini --limit 5

# Full benchmark, 1,251 questions, 5 MCQ rotations
viper-eval --model gpt-5.4-mini

Results land at eval_logs/<model>/<timestamp>/:

eval_logs/gpt-5.4-mini/<timestamp>/
├── results.json   # paper-aligned metrics + full provenance config
└── samples.jsonl  # one record per (question × rotation) trial

jq '{overall_score, mcq_accuracy, kprim_score, free_text_judge}' \
    eval_logs/gpt-5.4-mini/*/results.json

Step 1. Serve your model behind any OpenAI-compatible endpoint:

vllm serve my-org/my-vlm --port 8000

Step 2. Point viper-eval at it:

viper-eval \
    --model my-org/my-vlm \
    --api-base http://localhost:8000/v1 \
    --api-key dummy

Step 3. Read results.json. The numbers are directly comparable to the table below.

For working end-to-end examples, see examples/.

All numbers are mean accuracy (%) on n=1,251 questions. MCQ scores are means across 5 cyclic-shift rotations. The full table with 95% bootstrap confidence intervals is provided in the paper.

See docs/REPRODUCIBILITY.md for the exact viper-eval calls that reproduces each row.

viper-eval --help shows the full surface. The options most users will reach for:

VIPER is hosted on the Hugging Face Hub at MahmoodLab/viper. Machine-readable metadata in Croissant format is in croissant.json and on the Hub. The full datasheet is in docs/DATASHEET.md.

from datasets import load_dataset

ds = load_dataset("MahmoodLab/viper")["test"]
sample = ds[0]
sample["image"]          # PIL.Image.Image (1024 × 1024 RGB)
sample["question_type"]  # "mcq" | "kprim" | "free_text"
sample["organ"]          # one of 9 organ slugs
sample["category"]       # one of 7 paper categories

@inproceedings{weishaupt2026viper,
  title     = {VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology},
  author    = {Weishaupt, Luca and de Brot, Simone and Asin, Javier and Grau-Roma, Lloren\c{c} and Reitsam, Nic and Song, Andrew H. and Bang, Dongmin and Le, Long Phi and Kather, Jakob Nikolas and Mahmood, Faisal and Jaume, Guillaume},
  year      = {2026}
}

Code and data are released under CC BY-NC-ND 4.0. TG-GATEs is released under CC BY-SA 2.1 JP and MMO under CC BY-NC 4.0. See docs/DATASHEET.md for the full licensing discussion.

Questions, errata, or contributions: issues, or email Faisal Mahmood and Guillaume Jaume.