🤖[ICRA2026] RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matching

• Official PyTorch implementation for paper: "RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matching."
  

• [2025/09/17] The code of RoboEye is released! 🎉
• [2025/01/31] RoboEye is accepted by ICRA2026! 🎉

• Setup
• Download pretrained 2D Feature Extractor
• Download pretrained Robot 3D Retrieval Transformer
• Download Text Tokenizer
• Download our preprocessed json files
• Download our checkpoints
• Object Identification in ARMBench

First, clone this repository to your local machine, and install the dependencies.

conda create -n roboeye python=3.11 -y
conda activate roboeye
cd RoboEye
pip install -r requirements.txt

❗ You can modify the PyTorch version to suit your machine.

• BEiT3-Base: #layer=12; hidden=768; FFN factor=4x; #head=12; patch=16x16; #parameters: 222M

Download the model weights here and load, or:

model = VGGT()
_URL = "https://huggingface.co/facebook/VGGT-1B/resolve/main/model.pt"
model.load_state_dict(torch.hub.load_state_dict_from_url(_URL))

beit3.spm is the sentencepiece model used for tokenizing texts.

from transformers import XLMRobertaTokenizer
tokenizer = XLMRobertaTokenizer("/your_beit3_model_path/beit3.spm")

The dataset will be made publicly available upon acceptance of the paper.

The checkpoints will be made publicly available upon acceptance of the paper.

First extract positive and negative examples for the adapter-based training.

python armbench/ID.py --model 'beit3_base_patch16_224' --input_size 224 --task 'armbenchpick1extract' --batch_size 128 \
 --layer_decay 0.65 --lr 2e-4 --epochs 30 --warmup_epochs 3 --drop_path 0.2 --sentencepiece_model 'beit3.spm' \
 --data_path 'path/to/your/dataset' --output_dir '/your_output_path/' --log_dir '/your_log_path/' --weight_decay 0.05  \
 --save_ckpt_freq 1 --finetune 'path/to/ckpt/beit3_base_patch16_224.pth' -vggt_path 'path/to/ckpt/model.pt' --eval

• model specifics the name of model we use in this experiments.
• log_dir is the folder dir that stores the ouput log.
• task specifics using armbenchpick1extract for only use pre-pick images, armbench3t1extract for use both pre-pick and post-pick images.
• data_path is the folder dir that stores the datasets.
• finetune specifics the dir to pretrained weight of BEiT-3 model.
• vggt_path specifics the dir to pretrained weight of VGGT model.

Train the 3D keypoint-based retrieval matcher with adapter-based strategy.

python armbench/rerank.py --model 'vggt' --task 'rerank' --batch_size 128 \
 --layer_decay 0.65 --lr 5e-5 --epochs 30 --warmup_epochs 3 --drop_path 0.2 --sentencepiece_model 'beit3.spm' \
 --data_path 'path/to/your/dataset' --output_dir '/your_output_path/' --log_dir '/your_log_path/' --weight_decay 0.05  \
 --save_ckpt_freq 1 -vggt_path 'path/to/ckpt/model.pt'

• model specifics the name of model we use in this experiments.
• log_dir is the folder dir that stores the ouput log.
• task specifics using rerank for training with only use pre-pick images, rerank3t1 for training with both pre-pick and post-pick images.
• data_path is the folder dir that stores the datasets.
• vggt_path specifics the dir to pretrained weight of VGGT model.

Sample positive and negative examples for the MRR-driven 3D-awareness training.

python armbench/ID.py --model 'beit3_base_patch16_224' --input_size 224 --task 'armbenchpick1sample' --batch_size 128 \
 --layer_decay 0.65 --lr 2e-4 --epochs 30 --warmup_epochs 3 --drop_path 0.2 --sentencepiece_model 'beit3.spm' \
 --data_path 'path/to/your/dataset' --output_dir '/your_output_path/' --log_dir '/your_log_path/' --weight_decay 0.05  \
 --save_ckpt_freq 1 --finetune 'path/to/ckpt/beit3_base_patch16_224.pth' -vggt_path 'path/to/ckpt/model.pth' --eval

• model specifics the name of model we use in this experiments.
• log_dir is the folder dir that stores the ouput log.
• task specifics using armbenchpick1sample for only use pre-pick images, armbench3t1sample for use both pre-pick and post-pick images.
• data_path is the folder dir that stores the datasets.
• finetune specifics the dir to pretrained weight of BEiT-3 model.
• vggt_path specifics the dir to re-trained weight of VGGT model.

Train the 3D-feature-awareness module with MRR-driven 3D-awareness training.

python armbench/rerank.py --model 'vggt' --task 'classifier' --batch_size 256 \
 --layer_decay 0.65 --lr 1e-3 --epochs 30 --warmup_epochs 3 --drop_path 0.2 --sentencepiece_model 'beit3.spm' \
 --data_path 'path/to/your/dataset' --output_dir '/your_output_path/' --log_dir '/your_log_path/' --weight_decay 0.05  \
 --save_ckpt_freq 1 -vggt_path 'path/to/ckpt/model.pth'

• model specifics the name of model we use in this experiments.
• log_dir is the folder dir that stores the ouput log.
• task specifics using classifier for selective training objective.
• data_path is the folder dir that stores the datasets.
• vggt_path specifics the dir to re-trained weight of VGGT model.

Final inference of the two-stage paradigm.

python armbench/ID.py --model 'beit3_base_patch16_224' --input_size 224 --task 'armbenchpick1' --batch_size 128 \
 --layer_decay 0.65 --lr 2e-4 --epochs 30 --warmup_epochs 3 --drop_path 0.2 --sentencepiece_model 'beit3.spm' \
 --data_path 'path/to/your/dataset' --output_dir '/your_output_path/' --log_dir '/your_log_path/' --weight_decay 0.05  \
 --save_ckpt_freq 1 --finetune 'path/to/ckpt/beit3_base_patch16_224.pth' -vggt_path 'path/to/ckpt/model.pth' --eval

• model specifics the name of model we use in this experiments.
• log_dir is the folder dir that stores the ouput log.
• task specifics using armbenchpick1 for only use pre-pick images, armbench3t1 for use both pre-pick and post-pick images.
• data_path is the folder dir that stores the datasets.
• finetune specifics the dir to pretrained weight of BEiT-3 model.
• vggt_path specifics the dir to re-trained weight of VGGT model.

This repository is built using the VGGT, the BEiT, the BEiTv2, the BEiTv3, the CLIP, the open_clip, the Oscar, the DeiT repository, the timm library and many other inspiring works in the community.

This project is licensed under the license found in the LICENSE file in the root directory of this source tree. Microsoft Open Source Code of Conduct