Model Quantization

[tangqie-tq]

Hi

Thank you for your great work. I was wondering if you have any plan to release quantized model? 1.2B is still too big to run in edge devices

[weikaih04]

Hi, thanks for the suggestions! I'm not very familiar with quantization, but I'll ask the group for some help. Could you share the platform/device you want to deploy WildDet3D on?

[tangqie-tq]

Thanks for quick reply! I am currently working with DGX-spark and Jetson Thor. Both of these use a unified memory architecture, so VRAM isn't an issue. However, the raw compute power is a bit limited.

[weikaih04]

Hi, we just updated a mixed-precision FP16-based method with torch.compile to speed up inference by 3X. Let me know if it helps. I also tried TensorRT and Quant but unfornutately it didn't work for now as we have a lot of customized attentions

[tangqie-tq]

I tried running the model with bf16, and it does bring a noticeable speedup. At 640×480 resolution, I can get around ~2 FPS.

Unfortunately, I ran into another issue on Jetson Thor. Since Jetson Thor uses the newer sm110a architecture, torch.compile fails because the ptxas-blackwell bundled with Triton does not seem to recognize this architecture name:

ptxas-blackwell fatal: Value 'sm_110a' is not defined for option 'gpu-name'

So bf16 itself helps, but currently torch.compile is blocked by Triton's ptxas support for sm110a.

[weikaih04]

Hi, could you try upgrading to CUDA 13?