Incorrect TensorRT inference after ONNX export with dynamic shapes (batch size >= 1)

[DLegushev]

Description:
When exporting a YOLOX-S model to ONNX with dynamic input shapes, the subsequent conversion to TensorRT produces incorrect inference results for batch size >= 1.

Observed behavior:

• ONNX model exported with dynamic batch dimension
• TensorRT engine built from this ONNX model
• Inference results are incorrect for batch size >= 1
• Bounding boxes are corrupted, inconsistent, or mismatched to inputs

Expected behavior:

• TensorRT inference should produce correct bounding boxes for any batch size, consistent with PyTorch / static-shape ONNX results

Notes:

• Issue does not occur with static-shape ONNX export
• Suspected problem in ONNX export or dynamic shape handling during TensorRT conversion

[Avafly]

Thanks for the detailed description. I have currently set the batch size to 1 in both trtexec and the code configs, so only single image inference is supported.

I am confused why dynamic input shapes are producing incorrect results, as the model should accept any input size between 32 and 1280 (which worked correctly in my tests). Could you share the steps you used for model conversion?

By the way, implementing batch inference isn't a big task, and I plan to add batch inference support in a few weeks.

[DLegushev]

Hello!

Steps for model conversion:

1. Run tools/export_onnx.py with enable dynamic axes as you mentioned in README
2. Conversion via trtexec trtexec --onnx=yolox_s.onnx --shapes=images:2x3x768x384 --saveEngine=model_simplify.engine --exportProfile=model_gn_simplify.json --useSpinWait --separateProfileRun

Such steps produce incorrect results on tensorRT inference with batch size = 2.

[Avafly]

Thanks for the info.

The incorrect results are due to using fixed input dimensions of 2x3x768x384. This causes inference to produce wrong results if the input shape or batch size doesn't match exactly.

Try the following command instead. It enables dynamic shapes with a max batch size of 2.

trtexec --onnx=yolox_s.onnx --minShapes=images:1x3x32x32 --optShapes=images:1x3x768x384 --maxShapes=images:2x3x1280x1280 --saveEngine=model_simplify.engine --exportProfile=model_gn_simplify.json --useSpinWait --separateProfileRun

[Avafly]

By the way, I have updated the code to support batch inference. You can pull the latest changes and try running detect_batch with a dynamic engine model to see if it works as expected.