GeoGR: Spatial–Semantic Token Compression and Group Routing for Efficient 3D VLMs
Yongxin Zheng*,1, Hao Wu*,1,2, Fang Li1, Hu Fu1, Peiran Yin1,2, Xudong Wang1, Haozhe Hu1,2, Xinghao Chen1,3, Yunpu Ma4, Wei Zhang1, Xiaoyu Shen1,†
1 EIT-NLP Lab, Eastern Institute of Technology, Ningbo
2 Shanghai Jiao Tong University · 3 Hong Kong Polytechnic University
4 Munich Center for Machine Learning, Ludwig Maximilian University of Munich
* Equal contribution. † Corresponding author.
Contact: haowu.83@sjtu.edu.cn, xyshen@eitech.edu.cn
If you find this work useful for your research and applications, please consider citing:
@misc{zheng2026geogrspatialsemantictokencompression,
title={GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs},
author={Yongxin Zheng and Hao Wu and Fang Li and Hu Fu and Peiran Yin and Xudong Wang and Haozhe Hu and Xinghao Chen and Yunpu Ma and Wei Zhang and Xiaoyu Shen},
year={2026},
eprint={2610.13035},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.13035},
}- [TODO] Checkpoints will be released soon.
- [2026.10.12] The preprint is now published
- [2026.10.06] The code is now published!
- Spatial–semantic token compression. GeoSemZip consolidates observations across views into 3D voxel tokens. Semantic saliency selects dominant tokens, Morton-order sampling preserves spatial coverage, and residual merging aggregates unselected features into contextual anchors.
- Recoverable group routing. GroupRoute combines late entry and early exit with query-conditioned group-wise skipping. Skipped tokens retain their hidden states and can resume computation after regrouping at later anchors.
- Two complementary backbones. GeoGR supports both LLaVA-OneVision, which uses geometry to guide compression, and Video-3D LLM, which also incorporates explicit 3D positional information.
- Highlights
- Results
- Installation
- Data
- Training
- Evaluation
- Documentation
- Citation
- License
- Acknowledgments
The manuscript reports the following results relative to the dense post-trained models at approximately 10% retained LLM-side visual FLOPs:
| Backbone | Performance retained | LLM prefill speedup | Time-to-first-token speedup | Active visual tokens per LLM layer |
|---|---|---|---|---|
| LLaVA-OneVision-7B | 95.18% | 6.8× | 2.4× | 638 |
| Video-3D LLM-7B | 96.80% | 6.1× | 2.3× | 671 |
LLaVA-OneVision results cover ScanQA and SQA3D. Video-3D LLM results cover ScanQA, SQA3D, Scan2Cap, ScanRefer, and Multi3DRefer. Performance retention is the average relative to the dense baseline over the reported metrics; it is not an absolute accuracy score. Prefill speedup measures the LLM prefill phase; time to first token includes vision encoding and projector compression. These values are reported in the manuscript and may vary with hardware and runtime.
Use Linux, Bash, Python 3.10, and NVIDIA GPUs with CUDA support. Both backends share one geogr environment. Training uses BF16 and DeepSpeed ZeRO-3; the example launchers use four GPUs with a global batch size of 16.
git clone https://github.com/EIT-NLP/GeoGR.git
cd GeoGR
conda create -n geogr python=3.10 -y
conda activate geogr
python -m pip install --upgrade pip
python -m pip install -e "./algorithm/LLaVA-NeXT[train]" \
-e "./algorithm/Video-3D-LLM[train]" \
-e "./lmms-eval[video]"
export ENV_NAME=geogrInstall FlashAttention separately for the default training attention implementation, using a build compatible with your PyTorch and CUDA versions. The backend training extras specify PyTorch 2.1.2, torchvision 0.16.2, NumPy >=1.26.4,<2, and Transformers >=4.53,<4.54.
Both backends expose a llava Python package. The maintained training and evaluation launchers select the appropriate source directory for each backend, so both use the same environment. Run the commands below from the GeoGR repository root. If the launchers cannot locate conda, set CONDA_BASE=/absolute/path/to/miniconda3.
Download the compression-aware post-training annotations from EIT-NLP/GeoRG-67K-CAT and follow the dataset repository's instructions to extract them. The paper uses 66,940 PRISM-selected examples:
| Source | Original training pool | PRISM-selected examples |
|---|---|---|
| ScanRefer | 36,665 | 11,000 |
| Multi3DRefer | 43,838 | 13,151 |
| Scan2Cap | 36,665 | 11,000 |
| ScanQA | 26,515 | 7,955 |
| SQA3D | 79,445 | 23,834 |
| Total | 223,128 | 66,940 |
Place the five training JSON files in data/processed/subsets/ under this repository, matching the filenames in the data preparation guide. LLaVA-OneVision uses this layout by default. For Video-3D LLM, copy the training manifest, use absolute json_path entries, and set DATA_YAML as described in the guide. The launchers use the selected subsets directly.
Prepare the scene assets and benchmark data following the bundled preprocessing instructions. Copy the evaluation path template below and fill in your local data paths:
cp lmms-eval/scannet3d_data_paths.example.yaml \
lmms-eval/lmms_eval/tasks/_task_utils/scannet3d/data_paths.local.yaml
export THREE_D_CONFIG="$PWD/lmms-eval/lmms_eval/tasks/_task_utils/scannet3d/data_paths.local.yaml"Compression-aware post-training has two stages. Stage I adapts the projector and LLM to GeoSemZip; Stage II adapts the LLM to late entry and early exit. Group-wise skipping is enabled only during inference and requires no additional training.
Run the commands from the repository root after preparing the data. Activate geogr and configure the local vision weights, coordinate cache, and GPUs:
export ENV_NAME=geogr
# Replace xxx with your actual SigLIP checkpoint path.
export SIGLIP_MODEL_PATH=xxx
export VISION_TOWER="$SIGLIP_MODEL_PATH"
export SCANNET3D_COORDS_CACHE_ROOT=/absolute/path/to/scannet3d-coords-cache
export GPU_IDS=0,1,2,3
export OFFLINE_MODE=1For Video-3D LLM or a custom annotation location, set DATA_YAML to your training manifest and configure the scene roots as described in the data preparation guide.
Freeze the vision encoder and train the projector and full language model with projector compression enabled:
# Replace xxx with your actual checkpoint path.
MODEL_PATH=xxx \
OUTPUT_ROOT=/absolute/path/to/checkpoints/ov-stage1 \
bash bash-paper/ov-train/train_geosemzip.shFor Video-3D LLM, use the same environment, configure IMAGE_FOLDER, VIDEO_FOLDER, and EMBODIEDSCAN_FOLDER as described in the data preparation guide, and use:
# Replace xxx with your actual checkpoint path.
MODEL_PATH=xxx \
OUTPUT_ROOT=/absolute/path/to/checkpoints/video3d-stage1 \
bash bash-paper/video-3d-train/train_geosemzip.shStart from the Stage-I model, freeze the vision encoder and projector, and train the language model with the late-entry/early-exit window enabled. Despite its filename, train_grouproute.sh enables late_entry_early_exit during training; it does not enable group_wise_skip_recovery:
# Replace xxx with your actual Stage-I GeoSemZip checkpoint path.
MODEL_PATH=xxx \
OUTPUT_ROOT=/absolute/path/to/checkpoints/ov-stage2 \
bash bash-llm/ov-train/train_grouproute.shFor Video-3D LLM:
# Replace xxx with your actual Stage-I GeoSemZip checkpoint path.
MODEL_PATH=xxx \
OUTPUT_ROOT=/absolute/path/to/checkpoints/video3d-stage2 \
bash bash-llm/video3d-train/train_grouproute.shVideo-3D LLM also tunes its native 3D position embedding and grounding head by default. Both stages use one epoch, global batch size 16, learning rate 1e-5, cosine decay, warm-up ratio 0.03, and zero weight decay. Stage II requires a per-device batch size of 1; the launchers compute gradient accumulation to maintain the global batch size. See the paper protocol for the full configuration.
Evaluate the complete GeoGR pipeline with a Stage-II adapted checkpoint:
# LLaVA-OneVision: ScanQA and SQA3D.
# Replace xxx with your actual Stage-II late-entry/early-exit checkpoint path.
MODEL_PATH=xxx \
bash bash-llm/ILVAS/group-wise-skip-recovery/eval_geogr.sh
# Video-3D LLM: all five benchmarks; configure the Video3D data paths.
# Replace xxx with your actual Stage-II late-entry/early-exit checkpoint path.
MODEL_PATH=xxx \
bash bash-llm/ILVAS/video3d/eval_geogr.shRun the command for your selected backbone in the geogr environment. Override TASKS, GPU_IDS, OUTPUT_ROOT, or LIMIT as needed; for a small evaluation run, use LIMIT=1. Evaluation uses batch size 1 and greedy decoding with at most 512 new tokens.
The maintained projector settings are voxel_size=0.1, target_keep_ratio=0.30, dominant_ratio=0.85, max attention reduction, and grid_drop serialization. GroupRoute uses one-based inclusive visual windows [8,23] for LLaVA-OneVision and [8,24] for Video-3D LLM, anchors [9,13], and keep ratios [0.5,0.5]. Launcher exit boundaries are exclusive: EARLY_EXIT_LAYER=24 and 25, respectively.
For projector-only evaluation and comparison methods, see the LLaVA-OneVision evaluation scripts and Video-3D LLM evaluation scripts.
| Guide | Contents |
|---|---|
| Data preparation | Annotation placement, scene assets, and local paths |
| Reproduction | Environment variables, coordinate caches, and stage-by-stage commands |
| Compression pipeline | GeoSemZip and GroupRoute implementation semantics |
| Paper protocol | Benchmark splits, compression settings, and optimization |
| Repository layout | Model backends, evaluator, and launcher organization |
If you find GeoGR useful, please cite the manuscript. This provisional entry will be updated with the public paper identifier when available.
@misc{zheng2026geogr,
title = {GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs},
author = {Zheng, Yongxin and Wu, Hao and Li, Fang and Fu, Hu and Yin, Peiran and Wang, Xudong and Hu, Haozhe and Chen, Xinghao and Ma, Yunpu and Zhang, Wei and Shen, Xiaoyu},
year = {2026},
note = {Manuscript},
url = {https://github.com/EIT-NLP/GeoGR}
}This source snapshot includes upstream code with its own license terms: Video-3D LLM and LMMs-Eval.
We thank the authors and maintainers of LLaVA-OneVision / LLaVA-NeXT, Video-3D LLM, LMMs-Eval, and HiDrop, as well as the benchmark and compression-method authors whose work supports this project.
For questions, suggestions, or collaboration opportunities, please feel free to reach out:
- Hao Wu: haowu.83@sjtu.edu.cn
- Xiaoyu Shen: xyshen@eitech.edu.cn
-
Survey
- [Awesome-MLLM-Compression] From Data to Model: A Survey of the Compression Lifecycle in MLLMs
-
Vision Encoder
-
ImageLLM
- [EMNLP 2025] VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
- [ICLR 2026] HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
- [Preprint] ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention
- [Preprint] ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention