# Visual Inertial System (VINS) with Stereo Vision and GPU Acceleration

**Note: This tutorial is applicable for Kerloud SLAM Indoor only.**

The Kerloud SLAM Indoor set is equipped with a Nvidia TX2 module and an Intel Realsense D435i stereo camera. With a more powerful GPU core, the hardware set is capable of performing fully autonomous indoor localization with pure vision. The VINS system opens broad opportunities for vision-aided autonomy and other applications like SLAM, AR(Augmented Reality). The product is suitable for high-end users with the need to delve deep into computer vision and robot autonomy.

## Background

The VINS system is based on the original work from the HKUST Aerial Robotics Group (<https://uav.hkust.edu.hk>). The related resources can be found in the Reference section of this tutorial. The approach fuses the IMU data seamlessly with onboard computer vision, and can output real-time pose estimation. It provides a complete pipeline from front-end IMU preintegration to backend global optimization, as depicted in the figure below:

![](https://872799980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ml3VlSesh_3iSoHlURD%2Fuploads%2Fgit-blob-1f9df9e44dd97901fa5941e4e1b6d93b94b5d26a%2Fvins_pipeline.png?alt=media)

## What's Inside

Note that the original open-source code cannot be directly deployed on UAVs, and it requires sophisticated setup before a user can attempt a successful flight. The work we have done includes the following aspects:

* Complete environment setup with necessary prerequisites (OpenCV, Intel Realsense libraries, etc)
* Camera calibration with both intrinsic and extrinsic parameters
* Configurations for VINS fusion with Realsense stereo vision
* Integration with the autopilot for low-level state estimation
* Conduct dataset simulation, handhold move and autonomous flight tests

## How to Use

The workspace for the VINS system located in the directory \~/src/catkin\_ws\_VINS-Fusion-gpu contains following packages:

* VINS-Fusion-gpu: VINS algorithm package with GPU acceleration, customized specifically for Kerloud UAV.
* vison\_opencv: a package for interfacing ROS with OpenCV.
* pose\_converter: a package to bridge the odometry system with the autopilot.

The VINS system can be launched with following commands:

```
    # terminal 1
    roscore

    # terminal 2: launch mavros node
    cd ~/src/catkinws_realsense \
    && source devel/setup.bash \
    && roslaunch mavros px4.launch fcu_url:=/dev/ttyPixhawk:921600

    # terminal 3: launch realsense driver node
    cd ~/src/catkinws_realsense \
    && source devel/setup.bash \
    && roslaunch realsense2_camera rs_d435i_kerloud_stereo_slam.launch

    # terminal 4: launch VINS node
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && rosrun vins vins_node /home/ubuntu/src/catkin_ws_VINS-Fusion-gpu/src/VINS-Fusion-gpu/config/kerloud_tx2_d435i/realsense_stereo_imu_config.yaml

    # (Optional) terminal 5: launch vins-loop-fusion node
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && rosrun loop_fusion loop_fusion_node /home/ubuntu/src/catkin_ws_VINS-Fusion-gpu/src/VINS-Fusion-gpu/config/kerloud_tx2_d435i/realsense_stereo_imu_config.yaml

    # terminal 6: launch pose_converter node
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && roslaunch pose_converter poseconv.launch

    # (Optional) terminal 7: launch rviz for visualization
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && roslaunch vins vins_rviz.launch
```

or simply:

```
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && bash run.sh
```

## Demo

### Simulation with Dataset

The simulation test with Euroc dataset can be conducted with commands below:

```
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && roslaunch vins vins_rviz.launch \
    && rosrun vins vins_node ~/src/catkin_ws_VINS-Fusion-gpu/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml

    # Optional:
    cd ~/src/catkin_ws_VINS-Fusion-gpu \
    && source devel/setup.bash \
    && rosrun loop_fusion loop_fusion_node ~/src/catkin_ws_VINS-Fusion-gpu/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml \
    && rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag
```

![](https://872799980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ml3VlSesh_3iSoHlURD%2Fuploads%2Fgit-blob-e85a4ca870a70728b0d6262d143f34d6817ca76f%2Feuroc_sim.gif?alt=media)

### Ground Test

VINS hand-hold test:

![](https://872799980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ml3VlSesh_3iSoHlURD%2Fuploads%2Fgit-blob-5b9568c86ea3b18122d20cace769c91c13088cdf%2Fvins_handhold_short.gif?alt=media)

VINS dynamic ground test:

![](https://872799980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ml3VlSesh_3iSoHlURD%2Fuploads%2Fgit-blob-87e57a4ec5e08400c1b4b6db71b0cbdbe8e6be67%2Fvins_rovermove_short.gif?alt=media)

### Autonomous Indoor Hovering Flight

![](https://872799980-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-Ml3VlSesh_3iSoHlURD%2Fuploads%2Fgit-blob-d8adabe1f5ca8c8f09aa94ac00467d775d21ca55%2Fvins_hover.gif?alt=media)

## References

* VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, Tong Qin, Peiliang Li, Zhenfei Yang, Shaojie Shen, IEEE Transactions on Robotics. 2018.
* <https://github.com/HKUST-Aerial-Robotics/VINS-Mono>
* <https://github.com/HKUST-Aerial-Robotics/VINS-Fusion>