Synthetic Cookbook for Using/Testing/Demonstrating VisualDet3D in ROS
One of the most often required features of visual detection 3D is a full tool chain including data / model training / model testing / ROS deployment / demonstration and more.
In this cookbook, a set of tools are introduced to help this entire process.
It mainly involves three open-source repo:
Menu:
- Synthetic Cookbook for Using/Testing/Demonstrating VisualDet3D in ROS
- 1. Data
- 2. Training / Numerical Testing
- 3. Visualize offline test results (Optional)
- 4. Visualize real-time prediction results in ROS.
- 5. Visualize real-time streaming in ROS.
- 6. Streaming results on other/customized datasets
1. Data
Download the KITTI object dataset to local. The file structure should end up like /data/kitti/{training/testing}/{image_2/calib/label_2...}.
Please follow the setup process of kitti_visualize and make sure you could correctly visualize kitti object data (unselect isSequential in the GUI, and playing around with the index number).
Now you can play around with the data and also confirm the file structure is fine.
2. Training / Numerical Testing
Checkout the training pipeline in visualDet3D for mono3D.
If you do not want to train, one of the methods is to download the code and assets at Release 1.0 of visualDet3D to reproduce the result of a pre-uploaded checkpoint.
3. Visualize offline test results (Optional)
Checkout the additional labels part.
Basically, you need to copy the texts under workdirs/Yolo3D/output/testing/data of visualDet3D into /data/kitti_obj/testing/additional_label_2, and launch kitti_visualize on the testing split to visual the offline results.
4. Visualize real-time prediction results in ROS.
Checkout the setup process of visualDet3D_ros. Notice to modify the "visual3d_path", "cfg_file", "weight_path" parameters based on your own setting of visualDet3D.
Launch both kitti_visualize and visualDet3D_ros, unselect isSequential in the GUI. With the default launch file, the two node publish and read on the same image/camera_param topics. By adjusting what is shown in the RVIZ, we can now visualize the inference result in real-time.
5. Visualize real-time streaming in ROS.
We suggest download the KITTI raw dataset to streaming out image/lidar data.
-
Please follow the setup process of kitti_visualize and make sure you could correctly visualize kitti raw data (select isSequential in the GUI, and playing around with the index number).
-
Launch visualDet3D_ros as usual.
With the sequence data in kitti_visualize keep streaming into ROS, visualDet3D_ros will also streamingly conduct inference on the images, producing demo results like the one on the readme page of visualDet3D_ros.
Notice, that the IO/computational stress will be high at this step. You could adjust the streaming speed of kitti_visualize by changing local param in the launch file(UPDATE_FREQUENCY) or directly modifying the code.
6. Streaming results on other/customized datasets
After sucessfully setting up visualDet3D_ros, practically we can test the inference of visualDet3D on any image streams with ROS interface (like USB webcam).
We provide ROS interface to nuScenes dataset and KITTI-360 dataset.
To get more robust results, training on customized datasets, or trying camera-insensitive algorithms (like the adapted version of MonoFlex in visualDet3D) worth the efforts.
By adapting your own dataset to kitti/kitti360/nuscenes formats, you can also make use of these existing tools to boost development of your 3D detection project.