Z3D: Zero-Shot 3D Visual Grounding from Images

This repository contains an implementation of Z3D, a zero-shot method for 3d visual grounding introduced in our paper:

Z3D: Zero-Shot 3D Visual Grounding from Images
Nikita Drozdov, Andrey Lemeshko, Nikita Gavrilov, Anton Konushin, Danila Rukhovich, Maksim Kolodiazhnyi
https://arxiv.org/abs/2602.03361

1. Create the new virtual environment:

python -m venv .
source ./bin/activate 

2. Install Pytorch with CUDA support:

pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

3. Install SAM3:

cd sam3
pip install -e .
cd ..

4. Install requirements for LENS:

cd lens
pip install -r requirements.txt
cd ..

5. Install additional dependencies:

pip install -r requirements.txt

You also need to run a vLLM server to host the VLM agent. We recommend installing vLLM in a separate virtual environment to avoid dependency conflicts. Follow the official installation guide: https://docs.vllm.ai/en/latest/getting_started/installation/.

Our experiments were conducted with vllm==0.11.0.

Follow the instructions in the ./data folder to prepare the data.

Please request access to the checkpoints via the SAM 3 Hugging Face repository. Once your request is approved, download the SAM3 checkpoint and save it to ./sam3/pretrained/sam3.pt.

Download weights for LENS from Hugging Face:

# RefCOCO pretrained weights
huggingface-cli download --resume-download OuyBin/LENS --local-dir ./pretrained/qwen2p5_refcoco

# Weights before ReasonSeg fine-tuning
huggingface-cli download --resume-download OuyBin/LENS_ReasonSeg --local-dir ./pretrained/qwen2p5_refcoco_1500step

# ReasonSeg fine-tuninh weights
huggingface-cli download --resume-download OuyBin/LENS_ReasonSeg_FT --local-dir ./pretrained/qwen2p5_reasonseg_ft

# ReasonSeg CoT weights
huggingface-cli download --resume-download OuyBin/LENS_ReasonSeg_CoT --local-dir ./pretrained/qwen2p5_reasonseg_cot

Before running Z3D, start a VLM on a vLLM server. We use the following command for the 30B model:

vllm serve Qwen/Qwen3-VL-30B-A3B-Thinking \
  --tensor-parallel-size 2 \
  --allowed-local-media-path / \
  --port 8001 \
  --gpu-memory-utilization 0.95 \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0

To run Z3D inference on ScanRefer, use the following command:

PYTHONPATH=./sam3 python run_scanrefer.py --config configs/z3d_qwen30b_scanrefer.yaml

For inference on Nr3D, run the following command:

PYTHONPATH=./sam3 python run_nr3d.py --config configs/z3d_qwen30b_nr3d.yaml

In this setup, method uses ground-truth point clouds and Qwen/Qwen3-VL-30B-A3B-Thinking model as VLM agent. We also provide configuration files for inference on point clouds from posed images, from unposed images, as well as for other VLM agents.

We also evaluated LENS as segmentation backbone for Z3D. This model does not rely on a reasoning VLM and is therefore faster than SAM3-Agent. The segmentation backbone can be selected using the seg_model field in the onfiguration file. To run Z3D with LENS on the ScanRefer subset, use the following command:

PYTHONPATH=./lens python run_scanrefer.py --config configs/z3d_qwen30b_scanrefer_subset250_lens.yaml

If you find this work useful for your research, please cite our paper:

@article{drozdov2026z3d,
  title={Z3D: Zero-Shot 3D Visual Grounding from Images},
  author={Drozdov, Nikita and Lemeshko, Andrey and Gavrilov, Nikita and Konushin, Anton and Rukhovich, Danila and Kolodiazhnyi, Maksim},
  journal={arXiv preprint arXiv:2602.03361},
  year={2026}
}