Issues working with zed camera

[ArghyaChatterjee]

Hello, thanks for the work. There are couple of issues that I faced in this.

Description:

• Since zed is giving rectified images, so there is no distortion in the image. When I set the distortion model to none inside RightCameraParams.yaml and LeftCameraParams.yaml, I get the error saying F0110 11:51:41.113411 1214988 StereoCamera.cpp:375] Unknown DistortionModel: 0. Here is how it looks like:

LeftCameraParams.yaml

%YAML:1.0
# General sensor definitions.
camera_id: left_cam


# Sensor extrinsics wrt. the body-frame.
T_BS:
  cols: 4
  rows: 4
  data: [1.0, 0.0, 0.0, 0.0,
         0.0, 1.0, 0.0, 0.03,
         0.0, 0.0, 1.0, 0.013,
         0.0, 0.0, 0.0, 1.0]

# Camera specific definitions.
rate_hz: 30
resolution: [1920, 1080]
camera_model: pinhole
intrinsics: [1388.96728515625, 1388.96728515625, 954.34521484375, 531.0472412109375] #fu, fv, cu, cv
distortion_model: none
distortion_coefficients: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

I get the error which is attached with the console output.

• The imu in zed has a transform from the base_link frame. When I am trying to give the transform inside the ImuParams.yaml file, I get an error saying the extrinsics for the imu should be identity (which basically means I have to keep the imu to the base_link frame which I don't want to do).

ImuParams.yaml file looks like this:

# Sensor extrinsics wrt. the body-frame.
T_BS:
  cols: 4
  rows: 4
# THis is what I kept to get pass the error for now (but it's not correct)
  data: [1.0, 0.0, 0.0, 0.0,
         0.0, 1.0, 0.0, 0.0,
         0.0, 0.0, 1.0, 0.0,
         0.0, 0.0, 0.0, 1.0]

# Created by me but gives the above mentioned error:
# data: [0.99999000, -0.00199198, -0.00400396, -0.002,
  #        0.00200798,  0.99999000,  0.00399596,  0.015,
  #        0.00399596, -0.00400396,  0.99998400,  0.011,
  #        0.00000000,  0.00000000,  0.00000000,  1.000]
  
rate_hz: 30

Command:

$ roslaunch kimera_vio_ros kimera_vio_with_zedm.launch online:=true viz_type:=1 use_lcd:=false

Console output:

... logging to /home/arghya/.ros/log/71604292-cea3-11ef-a99f-a3a2610faa1d/roslaunch-arghya-Pulse-GL66-12UEK-1214885.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://127.0.0.1:42009/

SUMMARY
========

PARAMETERS
 * /kimera_vio_ros/kimera_vio_ros_node/base_link_frame_id: base_link
 * /kimera_vio_ros/kimera_vio_ros_node/bow_batch_size: 1
 * /kimera_vio_ros/kimera_vio_ros_node/bow_skip_num: 1
 * /kimera_vio_ros/kimera_vio_ros_node/force_same_image_timestamp: True
 * /kimera_vio_ros/kimera_vio_ros_node/image_transport: raw
 * /kimera_vio_ros/kimera_vio_ros_node/left_cam_frame_id: zedm_left_camera_...
 * /kimera_vio_ros/kimera_vio_ros_node/log_gt_data: True
 * /kimera_vio_ros/kimera_vio_ros_node/map_frame_id: map
 * /kimera_vio_ros/kimera_vio_ros_node/mono_ransac_threshold: 30
 * /kimera_vio_ros/kimera_vio_ros_node/odom_frame_id: odom
 * /kimera_vio_ros/kimera_vio_ros_node/online_run: True
 * /kimera_vio_ros/kimera_vio_ros_node/params_folder_path: /home/arghya/kime...
 * /kimera_vio_ros/kimera_vio_ros_node/position_det_threshold: 0.3
 * /kimera_vio_ros/kimera_vio_ros_node/publish_vlc_frames: True
 * /kimera_vio_ros/kimera_vio_ros_node/right_cam_frame_id: zedm_right_camera...
 * /kimera_vio_ros/kimera_vio_ros_node/robot_id: 0
 * /kimera_vio_ros/kimera_vio_ros_node/sensor_params_folder_path: 
 * /kimera_vio_ros/kimera_vio_ros_node/stereo_ransac_threshold: 20
 * /kimera_vio_ros/kimera_vio_ros_node/use_external_odom: False
 * /kimera_vio_ros/kimera_vio_ros_node/use_lcd: False
 * /kimera_vio_ros/kimera_vio_ros_node/use_lcd_registration_server: False
 * /kimera_vio_ros/kimera_vio_ros_node/use_rviz: True
 * /kimera_vio_ros/kimera_vio_ros_node/velocity_det_threshold: 0.1
 * /kimera_vio_ros/posegraph_viewer/frame_id: map
 * /rosdistro: noetic
 * /rosversion: 1.17.0
 * /use_sim_time: True

NODES
  /kimera_vio_ros/
    kimera_vio_ros_node (kimera_vio_ros/kimera_vio_ros_node)
    posegraph_viewer (pose_graph_tools_ros/visualizer_node)

ROS_MASTER_URI=http://127.0.0.1:11311

process[kimera_vio_ros/kimera_vio_ros_node-1]: started with pid [1214988]
process[kimera_vio_ros/posegraph_viewer-2]: started with pid [1214989]
[ INFO] [1736531498.520063794]: Initializing pose graph visualizer
[ INFO] [1736531498.649873687]: BoW vector batch size: 1
[ INFO] [1736531498.650465951]: BoW vector skip num: 1
[ INFO] [1736531498.650488763]: Publish VLC frames: 1
I0110 11:51:38.653298 1214988 Logger.cpp:59] Opening output file: traj_gt.csv
I0110 11:51:38.653398 1214988 RosOnlineDataProvider.cpp:43] Waiting for ROS time to be valid... 
(Sim Time is enabled; run rosbag with --clock argument)
F0110 11:51:41.113411 1214988 StereoCamera.cpp:375] Unknown DistortionModel: 0

*** Check failure stack trace: ***
    @     0x7f1a6ee1e1c3  google::LogMessage::Fail()
    @     0x7f1a6ee2325b  google::LogMessage::SendToLog()
    @     0x7f1a6ee1debf  google::LogMessage::Flush()
    @     0x7f1a6ee1e6ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f1a6ce6fdbc  VIO::StereoCamera::computeRectificationParameters()
    @     0x7f1a6ce711d5  VIO::StereoCamera::StereoCamera()
    @     0x7f1a6cfc30e1  VIO::StereoImuPipeline::StereoImuPipeline()
    @     0x7f1a6eba719d  VIO::KimeraVioRos::runKimeraVio()
    @     0x55b9865749c9  main
    @     0x7f1a6e1da083  __libc_start_main
    @     0x55b986574a4e  _start
================================================================================REQUIRED process [kimera_vio_ros/kimera_vio_ros_node-1] has died!
process has died [pid 1214988, exit code -6, cmd /home/arghya/kimera_vio_ws/devel/lib/kimera_vio_ros/kimera_vio_ros_node --use_lcd=false --vocabulary_path=/home/arghya/kimera_vio_ws/src/Kimera-VIO/vocabulary/ORBvoc.yml --flagfile=/home/arghya/kimera_vio_ws/src/Kimera-VIO/params/zedm/flags/Mesher.flags --flagfile=/home/arghya/kimera_vio_ws/src/Kimera-VIO/params/zedm/flags/VioBackend.flags --flagfile=/home/arghya/kimera_vio_ws/src/Kimera-VIO/params/zedm/flags/RegularVioBackend.flags --flagfile=/home/arghya/kimera_vio_ws/src/Kimera-VIO/params/zedm/flags/Visualizer3D.flags --use_external_odometry=false --do_coarse_imu_camera_temporal_sync=false --do_fine_imu_camera_temporal_sync=false --lcd_no_optimize=false --lcd_no_detection=false --depth_image_mask= --logtostderr=1 --colorlogtostderr=1 --log_prefix=1 --minloglevel=0 --v=0 --log_output=true --log_euroc_gt_data=true --output_path=/home/arghya/kimera_vio_ws/src/Kimera-VIO-ROS/output_logs/ --lcd_disable_stereo_match_depth_check=false --no_incremental_pose=false --viz_type=1 --visualize=true left_cam/image_raw:=/zedm/zed_node/left/image_rect_color left_cam/image_raw/compressed:=/zedm/zed_node/left/image_rect_color/compressed right_cam/image_raw:=/zedm/zed_node/right/image_rect_color right_cam/image_raw/compressed:=/zedm/zed_node/right/image_rect_color/compressed imu:=/zedm/zed_node/imu/data external_odom:=/zedm/zed_node/odom reinit_flag:=reinit_flag reinit_pose:=reinit_pose odometry:=odometry resiliency:=resiliency imu_bias:=imu_bias optimized_trajectory:=optimized_trajectory pose_graph:=pose_graph mesh:=mesh frontend_stats:=frontend_stats debug_mesh_img/image_raw:=debug_mesh_img/image_raw feature_tracks/image_raw:=feature_tracks/image_raw time_horizon_pointcloud:=time_horizon_pointcloud __name:=kimera_vio_ros_node __log:=/home/arghya/.ros/log/71604292-cea3-11ef-a99f-a3a2610faa1d/kimera_vio_ros-kimera_vio_ros_node-1.log].
log file: /home/arghya/.ros/log/71604292-cea3-11ef-a99f-a3a2610faa1d/kimera_vio_ros-kimera_vio_ros_node-1*.log
Initiating shutdown!

Additional files:
Please attach all the files needed to reproduce the error.

Please give also the following information:

• KimeraVIO branch, tag or commit used: master
• GTSAM version used: 4.4.0
• OpenGV version used: master
• OpenCV version used: 4.8
• Operating system and version : Ubuntu 20.04
• Did you change the source code? : no