Decrease ROS inference detections delay

[martincerven]

I'm using custom YOLOv8 models for detecting shuttlecocks and tracking them at 1280x800x30FPS, using simple PIDs that move motors, and I noticed there is big delay using larger models (~ 0.4s). Using small models, the delay is ~90ms, but I still think this could be improved. I tried:

• using --int8 quantization with tensorrt, but that made models completely unusable
• switching to rmw_zenoh_cpp: small improvements in some cases, could be faster if there's more cameras, etc
• training NN in D455's native 800x1200 resolution, but still I think the encoding/cropping take some time

What could work:

• I will try doing PR with latest YOLO version, maybe that could shave few ms off the delay
• also posting this to realsense ros repo could be useful
• not doing cropping
• more RMW tunning/testing
• using smallest, latest model, while getting okay accuracy
• doing QAT of the model, maybe that could work better with --int8 of tensorrt
• using movement model for motors that accounts for delay.

I believe having smallest delay as possible should be beneficial even for future Isaac ROS releases/versions, so it could be very useful to look more deeply into this.

Here are some ros2 logs:

admin@agx:/workspaces/isaac_ros-dev$ ros2 topic delay /camera_info_rect
WARNING: topic [/camera_info_rect] does not appear to be published yet
average delay: 0.084
        min: 0.033s max: 0.352s std dev: 0.09249s window: 27
average delay: 0.058
        min: 0.033s max: 0.352s std dev: 0.06823s window: 57
average delay: 0.050
        min: 0.033s max: 0.352s std dev: 0.05630s window: 87
average delay: 0.046
        min: 0.033s max: 0.352s std dev: 0.04903s window: 117
average delay: 0.043
        min: 0.033s max: 0.352s std dev: 0.04400s window: 147
average delay: 0.042
        min: 0.032s max: 0.352s std dev: 0.04025s window: 177
average delay: 0.041
        min: 0.032s max: 0.352s std dev: 0.03734s window: 207
average delay: 0.040
        min: 0.032s max: 0.352s std dev: 0.03497s window: 237
average delay: 0.039
        min: 0.032s max: 0.352s std dev: 0.03300s window: 267
average delay: 0.039
        min: 0.032s max: 0.352s std dev: 0.03133s window: 297
^Cadmin@agx:/workspaces/isaac_ros-dev$ ros2 topic delay /yolov8_encoder/resize/image
average delay: 0.063
        min: 0.053s max: 0.069s std dev: 0.00356s window: 28
average delay: 0.064
        min: 0.053s max: 0.069s std dev: 0.00304s window: 56
average delay: 0.064
        min: 0.053s max: 0.069s std dev: 0.00276s window: 86
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00389s window: 114
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00377s window: 143
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00354s window: 173
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00344s window: 202
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00342s window: 228
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00347s window: 257
average delay: 0.064
        min: 0.053s max: 0.089s std dev: 0.00342s window: 287
^Cadmin@agx:/workspaces/isaac_ros-dev$ ros2 topic delay /yolov8_encoder/crop/image
WARNING: topic [/yolov8_encoder/crop/image] does not appear to be published yet
average delay: 0.071
        min: 0.066s max: 0.075s std dev: 0.00251s window: 27
average delay: 0.071
        min: 0.066s max: 0.077s std dev: 0.00235s window: 57
average delay: 0.071
        min: 0.066s max: 0.077s std dev: 0.00227s window: 82
average delay: 0.072
        min: 0.066s max: 0.077s std dev: 0.00218s window: 107
average delay: 0.072
        min: 0.066s max: 0.077s std dev: 0.00211s window: 135
average delay: 0.072
        min: 0.065s max: 0.078s std dev: 0.00222s window: 165
average delay: 0.071
        min: 0.065s max: 0.078s std dev: 0.00218s window: 191
average delay: 0.071
        min: 0.065s max: 0.078s std dev: 0.00210s window: 220
average delay: 0.072
        min: 0.065s max: 0.078s std dev: 0.00209s window: 249
average delay: 0.072
        min: 0.065s max: 0.078s std dev: 0.00204s window: 277
^Cadmin@agx:/workspaces/isaac_ros-dev$ ros2 topic delay /detections_output
average delay: 0.092
        min: 0.083s max: 0.111s std dev: 0.00449s window: 29
average delay: 0.092
        min: 0.083s max: 0.111s std dev: 0.00343s window: 59
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00437s window: 89
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00435s window: 119
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00398s window: 149
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00370s window: 179
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00354s window: 209
average delay: 0.092
        min: 0.083s max: 0.120s std dev: 0.00367s window: 239
average delay: 0.092
        min: 0.082s max: 0.120s std dev: 0.00361s window: 269
average delay: 0.092
        min: 0.082s max: 0.120s std dev: 0.00354s window: 299
^Cadmin@agx:/workspaces/isaac_ros-dev$

[jaiveersinghNV]

Hi @martincerven ,

Thanks for trying Isaac ROS and sharing your experience! Cool project :)

As you correctly identified, there are many ways to improve the performance of model inference and your ROS graph. Before you begin, my recommendation is to use tools like ROS 2 Benchmark to measure the critical latency of each part of your graph. This will help you isolate the pre-processing, inference, and post-processing nodes.

We publish reference performance data for many Isaac ROS nodes and graphs here. If you observe a big discrepancy between your numbers and ours, then that might indicate a configuration or setup problem on your system.

If the performance data indicate that the custom DNN model itself is the bottleneck, then unfortunately we will not be able to offer support here. Consider modifying the model using some of the techniques you mentioned in your post, or upgrading to a more powerful NVIDIA Jetson device.