Multiple runs of yolo.predict() produce inconsistent results with GPU enabled

[nathankopp]

NOTE: This might be a problem underlying Larq and not necessarily with QuickYOLO specifically. I'm not sure.

Environment:

• quickyolov2 model
• Windows 10
• CUDA 11.2
• CUDNN 8.1.1.33
• Tensorflow 2.4.1
• Larq 0.11.2
• LCE 0.5.0

When GPU is enabled, calling yolo.predict() only works the first time. Subsequent calls produce invalid results. Sometimes the output tensor is filled with NaN values, while other times it contains garbage values. If I reload the model using Load_Yolo_model(), it works correctly again for one call to the predict() function.

evaluate_mAP.py shows a very low score, because only the first prediction produces valid results.
If I modify detection_demo.py to call detect_image() multiple times, only the first call works correctly.

If I change to os.environ["CUDA_VISIBLE_DEVICES"] = "-1" at the beginning of these files, the results are correct.

Note that training with the GPU does work correctly (apart from the call to evaluate_mAP).
Note also that this problem occurred prior to the commits on March 24.

[tehtea]

Thanks for your elaborate bug reports! It could be a TensorFlow / Larq / CUDA issue but as a workaround for now, os.environ["CUDA_VISIBLE_DEVICES"] = "-1" will be set at the beginning of evaluate_mAP.py and detection_demo.py.