Question about the depth sensor used for RGB-D inference

[tangqie-tq]

Hi,
I have a question regarding the use of depth information in WildDet3D. In the paper, it is mentioned that providing depth information and camera intrinsics can improve the model performance. However, in my own experiments, I observed a somewhat unexpected result.
I tested three input settings:
RGB only
RGB + camera intrinsics
RGB-D + camera intrinsics
I placed a person in front of the camera and compared the predicted height of the 3D bounding box. The person’s actual height is 1.78 m. The results were approximately:
RGB only: 1.75–1.80 m
RGB + intrinsics: 1.80–1.90 m
RGB-D + intrinsics: 1.50–1.60 m
Interestingly, the RGB-D setting produced a less accurate height estimate than RGB-only or RGB + intrinsics in my case.
I am using an Orbbec Gemini 336L RGB-D camera. I would like to ask what depth sensor or depth device was used in your experiments. Were there any specific requirements for the depth map, such as depth scale, alignment between RGB and depth, valid depth range, or preprocessing steps?
Any guidance would be greatly appreciated. Thank you!

RGB only

RGB + camera intrinsics

RGB-D + camera intrinsics

demo code

app.py

[weikaih04]

Hi, thanks for the careful experiments!

We trained the depth backend mainly on automotive LiDAR (Waymo / KITTI / Argoverse 2) and iPhone LiDAR (ARKitScenes) — no structured-light sensors, so Orbbec depth is somewhat out-of-distribution.

A quick diagnostic that would help us pinpoint the issue: the model returns its own predicted depth map as the last item in outputs (depth_maps[0], in meters). Could you run one frame twice — once monocular, once with depth_gt — and compare:

• D_pred_rgb vs D_raw (your Orbbec depth, at valid pixels inside the person bbox)
• D_pred_rgbd vs D_raw (same pixels)
• fraction of zero pixels inside the person bbox

That will tell us whether the sensor depth itself is inaccurate (in which case the monocular prediction would also disagree with the sensor) or whether the model just isn't adapting to this kind of depth input. Thanks!