Project Repository | Demo Video | English | 简体中文

FishLoop-VINS is a dual-fisheye visual-inertial localization system running on ROS 1. Building on fisheye VIO, the system adds loop-closure detection, geometric verification, and pose graph optimization for ultra-wide-angle cameras. It performs real-time state estimation using stereo fisheye images and IMU measurements, while multi-view loop-closure fusion reduces accumulated drift during long-term operation.

This repository provides complete support for dual-fisheye EUCM (Extended Unified Camera Model) cameras. Each fisheye camera can be unfolded into multiple canonical perspective views for feature tracking, loop-candidate retrieval, and matching. Candidate loop closures are then geometrically verified using the original EUCM pixels and bearing vectors, avoiding the incorrect treatment of fisheye images as ordinary pinhole images.

Watch the demo video

Using the Docker environment provided in this repository is recommended to avoid version conflicts among ROS, OpenCV, CUDA, and Ceres. This repository currently supports only devices equipped with an NVIDIA GPU.

• Ubuntu 20.04
• ROS Noetic
• NVIDIA GPU with a compatible host driver
• Docker
• NVIDIA Container Toolkit
• CUDA 11.8
• OpenCV 4.5.5 + CUDA
• Ceres Solver, Eigen3, Boost, and OpenMP
• libSGM 3.0.0 for the CUDA depth-estimation path

NVIDIA Container Toolkit must be installed before running the system.

git clone https://github.com/luohongk/FishLoop-VINS.git
cd FishLoop-VINS

The default GPU architecture is Ada 8.9, which is suitable for the RTX 40 series:

docker compose --profile gpu build

To support another GPU, set CUDA_ARCH and CUDA_ARCH_PTX according to the following table:

For example, use the following command for the Ampere architecture:

CUDA_ARCH=8.6 CUDA_ARCH_PTX=8.6 docker compose --profile gpu build

docker pull luohongkun0715/fishloop_vins:gpu
docker tag luohongkun0715/fishloop_vins:gpu fishloop_vins:gpu

Allow the container to access the host X11 server:

xhost +local:root

Mount the repository and data directories into the container:

docker run -it --rm \
  --gpus all \
  --network=host \
  --privileged \
  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
  -e DISPLAY="$DISPLAY" \
  -e QT_X11_NO_MITSHM=1 \
  -v /path/to/FishLoop-VINS:/root/catkin_ws/src/fishloop_vins \
  -v /path/to/data:/data \
  --name fishloop_vins_gpu \
  -w /root/catkin_ws \
  fishloop_vins:gpu

Replace the following paths:

• /path/to/FishLoop-VINS: the absolute path to the FishLoop-VINS repository
• /path/to/data: the absolute path to the directory containing the rosbag data

Example command used in the development environment for this repository:

xhost +local:root && docker run -it --rm \
  --gpus all \
  --network=host \
  --privileged \
  -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
  -e DISPLAY="$DISPLAY" \
  -e QT_X11_NO_MITSHM=1 \
  -v /home/lhk/workspace/FishLoop-VINS:/root/catkin_ws/src/fishloop_vins \
  -v /home/lhk/data:/data \
  --name fishloop_vins_gpu \
  -w /root/catkin_ws \
  fishloop_vins:gpu

The GPU Dockerfile provides all required dependencies, but the mounted source code must still be compiled in the container:

cd /root/catkin_ws
source /opt/ros/noetic/setup.bash
catkin_make -DCMAKE_POLICY_VERSION_MINIMUM=3.5 -j8
source devel/setup.bash

Continue in the current container terminal:

cd /root/catkin_ws
source devel/setup.bash
roslaunch fishloop_vins vins_fisheye_loop.launch

By default, this launch file starts the following components:

• fishloop_vins_node_fisheye: dual-fisheye VIO
• loop_fusion_node: loop-closure detection and pose graph optimization
• RViz: trajectory, loop-closure, and map visualization

Sample data: Google Drive: example.bag

Open another terminal on the host, enter the same container, and play the data:

docker exec -it fishloop_vins_gpu bash
rosbag play /data/example.bag --clock -r 0.3

Default output directory:

/root/catkin_ws/src/fishloop_vins/data

Because the repository is mounted into the container, the generated files also appear in the repository's data/ directory on the host.

vio.csv field format:

timestamp_ns,px,py,pz,qw,qx,qy,qz,vx,vy,vz

vio_loop.csv field format:

timestamp_ns,px,py,pz,qw,qx,qy,qz

FishLoop-VINS is based on and inspired by the following excellent open-source projects:

• VINS-Fusion: the foundational framework for visual-inertial estimation and loop fusion
• VINS-Fisheye: the foundational fisheye VINS implementation
• DBoW2: bag-of-words-based place recognition
• Ceres Solver: nonlinear least-squares optimization
• libSGM: CUDA-accelerated semi-global stereo matching

The LICENSE file in the repository root declares the project under the Apache License 2.0. Note that some ROS package metadata and source files inherited or modified from upstream projects may specify different licenses. Before public distribution or commercial use, verify and align the project's license metadata and comply with the license requirements of all third-party components.