[CVPR 2026] MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation

🎉 [News] Our paper has been accepted to CVPR 2026! 🎉

This repository is the official implementation of MVGGT. All resources have been fully released. We warmly welcome everyone to try out our code, models, and the interactive demo!

• Release the MVRefer Benchmark
• Release Training & Inference Code.
• Release Pre-trained Models.
• Release Interactive Demo Code (Local version).

Most existing 3D referring expression segmentation (3DRES) methods rely on dense, high-quality point clouds, while real-world agents such as robots and mobile phones operate with only a few sparse RGB views and strict latency constraints.

We introduce Multi-view 3D Referring Expression Segmentation (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images. Traditional two-stage pipelines, which first reconstruct a point cloud and then perform segmentation, often yield low-quality geometry, produce coarse or degraded target regions, and run slowly.

We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning. Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives.

Figure 1: Comparison of the proposed MV-3DRES task (bottom) against the traditional two-stage pipeline (top).

We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an end-to-end framework designed for efficiency and robustness.

Figure 2: Architecture of MVGGT. It features a Frozen Reconstruction Branch (top) and a Trainable Multimodal Branch (bottom).

Note: For interactive 3D visualizations and video comparisons with other methods, please visit our Project Page.

git clone https://github.com/sosppxo/mvggt.git
cd mvggt

Create and activate a new conda environment:

conda create -n mvggt python=3.12
conda activate mvggt

Install the full requirements for training:

pip install -r requirements.txt

Download the ScanNet dataset. The data should be organized as follows:

[data_root]/
|   └──scene0000_00/
│       ├── color/              # RGB images (.jpg)
│       ├── depth/              # Depth maps (.png)
│       ├── intrinsic/          # intrinsic_depth.txt
│       └── pose/               # Camera poses (.txt)
└── scans/                  # Required for 2D instance masks
    └── scene0000_00/
        └── scene0000_00_2d-instance-filt/
            └── instance-filt/  # 2D instance segmentation masks (.png)

⚠️ Note: Remember to update the data_root path for both train_dataset and test_dataset in configs/data/example.yaml to point to your actual [data_root].

Download ScanRefer annotations.

Put the ScanRefer folder in data/:

data/
└── ScanRefer/
    ├── ScanRefer_filtered_train.json
    ├── ScanRefer_filtered_val.json
    ├── ScanRefer_filtered_train.txt
    └── ScanRefer_filtered_val.txt

If you need to regenerate the invalid frame list based on your data, run:

python scripts/generate_invalid_scannet_list.py

To enable target-centric sampling and ensure the model sees the referred objects during training, we use pre-computed instance-to-frame mapping:

data/
└── scene_frame_indices/
    └── [scene_id].json

MVrefer benchmark contains frame selections for evaluation:

data/
└── mvrefer_val.json

For training and inference, you need to prepare the following weights in the ckpts/ directory:

The multimodal branch is initialized from Pi3. Download and place it in:

ckpts/
└── Pi3/

The model uses RoBERTa-base as the text encoder. Download and place it in:

ckpts/
└── roberta-base

Download the final MVGGT checkpoint from Hugging Face and update train.resume in eval_mvggt.sh.

To start training on ScanRefer:

bash train_mvggt.sh

Update the checkpoint path (train.resume) in eval_mvggt.sh, then run inference:

bash eval_mvggt.sh

Follow these steps to deploy the interactive demo locally:

Install the required packages for the demo:

pip install -r requirements_demo.txt

1. Download pre-trained model weights: Download from Hugging Face and update the ckpt_path in demo_gradio.py (line 608) to point to your checkpoint file.

2. Download RoBERTa tokenizer: The demo requires RoBERTa tokenizer. Download it using:

mkdir ckpts
python -c "from transformers import RobertaTokenizer; RobertaTokenizer.from_pretrained('roberta-base').save_pretrained('./ckpts/roberta-base')"

Or manually download from Hugging Face and place it in ./ckpts/roberta-base/.

Run the Gradio demo:

python demo_gradio.py

The demo will be available at http://localhost:7860.

1. Upload multiple images or a video containing multi-view scenes
2. Enter a referring expression describing the target object
3. The model will generate 3D segmentation results that can be downloaded as GLB files

If you find our work useful in your research, please consider citing:

@misc{wu2026mvggt,
  Author = {Changli Wu and Haodong Wang and Jiayi Ji and Yutian Yao and Chunsai Du and Jihua Kang and Yanwei Fu and Liujuan Cao},
  Title = {MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation},
  Year = {2026}
}