This repository contains the official implementation of the paper:
Revisit Self-supervised Depth Estimation with Local Structure-from-Motion
Authors: Shengjie Zhu and Xiaoming Liu
ECCV'24 📄 [arXiv], 🌐 [webpage]
This work studies a multiview-RANSAC scheme to find multi-view poses that maximize inliers. We further investigate whether such poses can improve state-of-the-art supervised depth map performance in a self-supervised manner by employing a NeRF as a triangulation machine.
git clone https://github.com/ShngJZ/SSfM
cd SSfM
git submodule update --init --recursive --remote
conda create -n ssfm python=3.10
conda activate ssfm
# Install PyTorch (https://pytorch.org/)
# Check and select the version compatible with your CUDA (check your CUDA version with nvcc --version)
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu121
# Install packages
pip install -r requirements.txt
# Install cupy according to your CUDA version, for CUDA 12.x:
pip install cupy-cuda12x
pip install timm==0.6.7 # ZoeDepth requirement
# Install LightedDepth
cd third_party/LightedDepth/GPUEPMatrixEstimation/
python -c "import torch; print(f'PyTorch CUDA version: {torch.version.cuda}')" && nvcc --version | grep 'Cuda compilation tools' # ensure the version of Pytorch and system nccc are the same
python setup.py install
# Download pretrained depth and correspondence checkpoints
mkdir checkpoint && cd checkpoint
gdown 1nOpC0MFWNV8N6ue0csed4I2K_ffX64BL # PDCNet
wget https://github.com/isl-org/ZoeDepth/releases/download/v1.0/ZoeD_M12_NK.pt # ZoeDepthIf you are installing a higher CUPY version (e.g., cupy 12x), you need to modify the PDC Net codebase. Update the following files:
third_party/DenseMatching/models/modules/local_correlation/correlation.pythird_party/DenseMatching/third_party/GOCor/GOCor/local_correlation/correlation.py
Replace the following code:
# Old code
@cupyutil.memoize(for_each_device=True)
def cupy_launch(strFunction, strKernel):
return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
# New code
@cupyutil.memoize(for_each_device=True)
def cupy_launch(strFunction, strKernel):
module = cupy.RawModule(code=strKernel, name_expressions=[strFunction])
return module.get_function(strFunction)Please ensure you agree to all licenses of the dataset before downloading.
# Download datasets
mkdir SSfM-Datasets && cd SSfM-Datasets
# ScanNet dataset, make sure the git lfs installed
git clone https://huggingface.co/datasets/shngjz/f2509922c5dc01d3c0f9a365327c1de3
mv f2509922c5dc01d3c0f9a365327c1de3 ScanNetNote: You may be prompted to enter your user ID and token multiple times. Please expect delays when downloading large datasets.
The main optimization process consists of two steps:
- Camera pose estimation
- Depth map optimization using NeRF
For multi-GPU training (example with 8 GPUs), run one command per Tmux window:
# Replace /home/ubuntu/disk1/RSfM-Datasets/ScanNet with your dataset path
CUDA_VISIBLE_DEVICES=0 python local_ba_scale_up/optimize_pose_depth.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --data_root /home/ubuntu/disk1/SSfM-Datasets/ScanNet --dataset scannet
# Repeat for CUDA_VISIBLE_DEVICES=1 through 7 on your tmux window# Compare to initial Mono-Depth inputs:
python evaluation/eval_nerf_depth.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --dataset scannet# Evaluate using GT depth map to compare consistent 3D points
python evaluation/eval_pose_by_3dcons.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --dataset scannet
# Evaluate using GT depth map to compare inlier correspondences
python evaluation/eval_pose_by_corres.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --dataset scannet
# Evaluate using predicted depth map to compare consistent depth maps
python evaluation/eval_pose_by_consisdepth.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --dataset scannet
# Evaluate using GT poses
python evaluation/eval_pose_by_pose.py --train_module joint_pose_nerf_training/scannet_depth_exp --train_name zoedepth_pdcnet --train_sub 5 --dataset scannetIf you find this work useful in your research, please consider citing:
@article{zhu2024revisit,
title={Revisit Self-supervised Depth Estimation with Local Structure-from-Motion},
author={Zhu, Shengjie and Liu, Xiaoming},
journal={ECCV},
year={2024}
}