Zhenhua Du · Binbin Xu · Haoyu Zhang · Kai Huo · Shuaifeng Zhi
MOSE is a NeRF-based semantic reconstruction pipeline that lifts noisy monocular priors (2D semantics, segment masks, normals) into a neural field to jointly recover geometry and semantics in both 3D and 2D.
We use uv to manage the Python environment:
pip install uv
uv syncThe uv sync command creates a virtual environment under .venv and installs all
dependencies specified in pyproject.toml.
ScanNet Scenes
- Download the ScanNetv2 scenes and update the dataset path in the config file.
- Convert the raw scenes to the MOSE format:
python exp_preprocess.py --data_type scannet
- We select the following scenes for evaluation:
scene0050_00,scene0616_00,scene0378_00,scene0435_02,scene0084_00,scene0426_00,scene0025_00, andscene0169_00.
Monocular Priors
- Semantic priors: we employ DeepLabV3+ with a ResNet-101 backbone to generate 2D semantic predictions. The checkpoint can be downloaded here
- Segmentation priors: run
preprocess/preprocess_spp.pyfor superpixel priors andpreprocess/preprocess_sam.pyfor SAM priors (stored undergrids/). - Normal priors: follow the NeuRIS preprocessing steps described here.
Data Organization
Each scene directory should follow the structure below:
<scene_name>
|-- cameras_sphere.npz # camera intrinsics/extrinsics
|-- image/train/0000.png # RGB images (train/test)
|-- depth/train/0000.png # metric depth maps
|-- pose/train/0000.txt # camera-to-world poses
|-- normal/pred_normal/0000.npz # normal priors
|-- semantic
| |-- semantic_GT/0000.png
| |-- predicted_semantic/0000.png
|-- grids
| |-- sam/0000.png # SAM segments
| |-- spp_seg/0000.png # superpixels
|-- xxx.ply # ground-truth mesh
|-- xxx.labels.ply # ground-truth semantic mesh
|-- trans_n2w.txt # normalization-to-world transform
python exp_runner.py --mode train --conf ./confs/demo.conf --scene_name scene0050_00python exp_runner.py --mode validate_mesh --conf ./confs/demo.conf --is_continueThis generates both geometric and semantic meshes.
python exp_runner.py --mode validate_image --conf ./confs/demo.conf --is_continuepython exp_evaluation.py --mode eval_3D_mesh_TSDF --exp_name demo
python exp_evaluation.py --mode eval_semantic_3D --exp_name demo
python exp_evaluation.py --mode eval_semantic_2D --exp_name demo- Our work is built on NeuRIS and Manhattan-SDF.
- Parts of the code are adapted from Semantic-NeRF and MonoSDF.
@article{du2024mose,
title = {MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors},
author = {Du, Zhenhua and Xu, Binbin and Zhang, Haoyu and Huo, Kai and Zhi, Shuaifeng},
journal = {IEEE Robotics and Automation Letters},
year = {2024},
volume = {9},
number = {11},
pages = {10343--10350},
publisher = {IEEE}
}