This repository provides the code for the paper "Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection".
Install your Python environment (recommended version=3.8) and BLIP-2.
cd LAVIS-modified
pip install -e .
HICO-DET dataset can be downloaded here. After finishing downloading, unpack the tarball (hico_20160224_det.tar.gz) to the data directory.
Instead of using the original annotations files, we use the annotation files provided by the PPDM authors. The annotation files can be downloaded from here. The downloaded annotation files have to be placed as follows.
data
└─ hico_20160224_det
|─ annotations
| |─ trainval_hico.json
| |─ test_hico.json
| └─ corre_hico.npy
:
First clone the repository of V-COCO from here, and then follow the instruction to generate the file instances_vcoco_all_2014.json. Next, download the prior file prior.pickle from here. Place the files and make directories as follows.
BC-HOI
|─ data
│ └─ v-coco
| |─ data
| | |─ instances_vcoco_all_2014.json
| | :
| |─ prior.pickle
| |─ images
| | |─ train2014
| | | |─ COCO_train2014_000000000009.jpg
| | | :
| | └─ val2014
| | |─ COCO_val2014_000000000042.jpg
| | :
| |─ annotations
: :
For our implementation, the annotation file have to be converted to the HOIA format. The conversion can be conducted as follows.
PYTHONPATH=data/v-coco \
python convert_vcoco_annotations.py \
--load_path data/v-coco/data \
--prior_path data/v-coco/prior.pickle \
--save_path data/v-coco/annotations
Note that only Python2 can be used for this conversion because vsrl_utils.py in the v-coco repository shows a error with Python3.
V-COCO annotations with the HOIA format, corre_vcoco.npy, test_vcoco.json, and trainval_vcoco.json will be generated to annotations directory.
Download the pretrained model of DETR detector for ResNet50, and put it to the params directory.
python ./tools/convert_parameters.py \
--load_path params/detr-r50-e632da11.pth \
--save_path params/detr-r50-pre-2branch-hico.pth \
--num_queries 64
python ./tools/convert_parameters.py \
--load_path params/detr-r50-e632da11.pth \
--save_path params/detr-r50-pre-2branch-vcoco.pth \
--dataset vcoco \
--num_queries 64
After the preparation, you can start training with the following commands.
bash exp/train_hico.sh
Using a single GPU, Modify --pretrained and add --eval in the corresponding sh file for inference
add --zero_shot_type xxxx --fix_clip --del_unseen for UC/UO/UV testing as GEN-VLKT (https://github.com/YueLiao/gen-vlkt/tree/master/configs)
add --KO for Know_Object testing
for training.
bash exp/train_vcoco.sh
Firstly, you need the add the following main function to the vsrl_eval.py in data/v-coco.
if __name__ == '__main__':
import sys
vsrl_annot_file = 'data/vcoco/vcoco_test.json'
coco_file = 'data/instances_vcoco_all_2014.json'
split_file = 'data/splits/vcoco_test.ids'
vcocoeval = VCOCOeval(vsrl_annot_file, coco_file, split_file)
det_file = sys.argv[1]
vcocoeval._do_eval(det_file, ovr_thresh=0.5)
Next, for the official evaluation of V-COCO, a pickle file of detection results have to be generated. You can generate the file with the following command. and then evaluate it as follows.
bash exp/eval_vcoco.sh
cd data/v-coco
python vsrl_eval.py vcoco.pickle
BC-HOI contains an opt model, resulting in a file that is too large and there are no plans to upload it at the moment
If you find this project useful in your research, please consider citing our paper:
@inproceedings{hu2025bilateral,
title={Bilateral collaboration with large vision-language models for open vocabulary human-object interaction detection},
author={Hu, Yupeng and Ding, Changxing and Sun, Chang and Huang, Shaoli and Xu, Xiangmin},
booktitle={Proceedings of the IEEE/CVF international conference on computer vision},
pages={20126--20136},
year={2025}
}