Evaluation and data-preparation toolkit for the BOP-Refer benchmark — language-grounded 2D and 3D object localization from natural-language referring expressions and a single RGB image.
pip install -e ".[dev]"| Module | Purpose |
|---|---|
bop_refer/eval/ |
Evaluation pipeline (2D and 3D tracks). |
bop_refer/dataprep/ |
Data preparation (download BOP datasets, compute OBBs, build objects_info). |
bop_refer/vis/ |
Visualization (render meshes with OBB wireframes, compile PDFs). |
data_generation/ |
Query generation pipeline (render + describe objects, LLM queries, verification, final dataset). |
vlm-evals/ |
VLM benchmarking harness (15 models, parallel runs, prompt ablations). |
See docs/bop_refer_data_format.md
for the full specification.
A BOP-Refer data bundle has this layout:
bop_refer_data/
├── objects_info.parquet # 246 objects, symmetries, model-frame OBBs
├── images_test/
│ ├── shard-000000.tar # WebDataset image shards
│ └── shard-000001.tar
├── images_info_test.parquet # Per-image metadata + intrinsics
├── queries_test.parquet # Referring expressions
└── gts_test.parquet # Ground-truth 2D + 3D bounding boxes
First prepare your predictions in this format. Then you can compute metrics for the 2D/3D tasks using:
python -m bop_refer.eval.evaluate \
--gts-path gts_test.parquet \
--preds-2d-path preds_2d.parquet \
--preds-3d-path preds_3d.parquet \
--objects-info-path objects_info.parquet \
--output eval_results.jsonEither --preds-2d-path or --preds-3d-path can be omitted to skip that
track. The --objects-info-path provides per-object symmetry transforms
used for symmetry-aware 3D IoU computation.
# Download everything for all datasets.
python -m bop_refer.dataprep.download_bop_datasets
# Download only models and test images for specific datasets.
python -m bop_refer.dataprep.download_bop_datasets \
--datasets ycbv tless \
--modalities models testComputes a tight oriented bounding box (OBB) for each object mesh. Box orientation depends on symmetry type (continuous, discrete, or none).
python -m bop_refer.dataprep.compute_model_bboxes \
--models-root bop_datasets \
--models-subdir models_eval \
--output model_bboxes.jsonAssembles per-object metadata from BOP models_info.json files and
precomputed OBBs.
python -m bop_refer.dataprep.create_objects_info \
--models-root bop_models \
--models-subdir models_eval \
--bboxes-json model_bboxes.json \
--output objects_info.parquetConverts images and GT annotations from BOP format to BOP-Refer format.
python -m bop_refer.dataprep.convert_bop_images \
--bop-root bop_datasets \
--split test \
--objects-info objects_info.parquet \
--images-csv selected_images.csv \
--output-dir bop_refer_dataRefer to data_generation/README.md for the
full LLM-based query generation pipeline.
# Render objects with OBB wireframes
python -m bop_refer.vis.visualize_objects \
--objects-info objects_info.parquet \
--models-root bop_models \
--models-subdir models \
--output-dir vis_output
# Compile images into PDF
python -m bop_refer.vis.compile_pdf_from_images \
--input-dir vis_output \
--output vis_output.pdfRefer to vlm-evals/README.md for the full
benchmarking harness supporting 15 vision-language models.
Quick example:
cd vlm-evals/
# Run Gemini 3.1 Pro with optimal prompt config
python run_gemini.py --runs gemini_31_pro_proposed_v2 \
--depth raw --few-shot 5 --workers 4 \
--data-dir ./data
# Check scores
cat outputs/gemini_31_pro_proposed_v2_5shot_raw/results.mdSee vlm-evals/llm-prompt-mapping.json for the optimal prompt
configuration per model.
- All metric scripts average LM and LMO scores into a single LM entry. The final AP3D is the macro-average over the 9 resulting dataset scores.
- 3D bounding boxes use millimeters in the OpenCV camera frame (X right, Y down, Z forward).
- 2D bounding boxes use
[xmin, ymin, xmax, ymax]in pixels.