Custom YOLOv11 digit detection model crashes with TIE728 acceleration on ESP32-S3, works slowly without it

[zongxiongw]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate.
• Provided a clear description of your suggestion.
• Included any relevant context or examples.

Issue or Suggestion Description

Hi,

I'm deploying a custom digit detection model (0-9 classes) on an ESP32-S3 (N8R8) using the official yolo11_detect example, and I'm running into a persistent issue with TIE728 acceleration.

Environment:

ESP-IDF: v6.0.1

ESP-DL: tried both v3.3.2 and v2.1.0

Chip: ESP32-S3 (N8R8, Octal PSRAM 8MB enabled)

Model: espdet_pico (0.36M parameters, input 224x224), trained on SVHN dataset with esp-detection, quantized with esp-ppq to .espdl

What I did:

Cloned the official yolo11_detect example (espressif/esp-dl=3.3.2:yolo11_detect).

Replaced the model file with my own .espdl.

Enabled PSRAM (Octal, 8MB) via menuconfig.

Built and flashed.

What happens:

With ESP-DL v3.3.2, the program crashes immediately on inference with:
Guru Meditation Error: Core 0 panic'ed (LoadProhibited) at tie728_s8_conv2d_h_w_unaligned_c_n_activation_loop0_no_preload_bias.

With ESP-DL v2.1.0, the model works correctly (detects digits) but inference takes ~26 seconds per image, which is far from the expected 7–8 FPS.

What I've tried:

Manually modified dl_base_conv2d.cpp to force C-only path (i_impl_func = nullptr), which avoids the crash but keeps performance very slow because TIE728 is completely bypassed.

Locking esp-dl to v2.1.0 works reliably but with unacceptable speed.

I've verified that the official bus.jpg test image works in both cases, but custom digit images also work slowly.

My questions:

Is there a known compatibility issue between custom espdet_pico models and the TIE728 assembly kernels? What causes the LoadProhibited crash?

Is there a recommended quantization setting, model export parameter, or runtime configuration that would allow me to safely enable TIE728 and achieve the expected ~7 FPS?

Alternatively, is there a way to keep the latest ESP-DL but selectively disable only the problematic TIE728 kernel while retaining other optimizations?

Any guidance or workaround would be greatly appreciated. I can provide the .espdl model file or the training pipeline if needed.

[Ginosko-mia]

Hi, @zongxiongw

1. I tested with espdet_pico + (nc=10) on the ESP-DL v3.3.2 and could not reproduce your issue. So far, I have not found any compatibility problem between custom espdet_pico models and the TIE728 assembly kernels.
   PS: using ESP-IDF: v6.0.1, ESP32-S3 (N8R8, Octal PSRAM 8MB enabled), espdet_pico+(nc=10)+ random weight

2. Due to differences in the DFL module design, please do not replace the model with your own .espdl model in the yolo11_detect example. You can try cat_detect instead.

3. The ~26 ms result is caused by model replacement failure. The default model in yolo11_detect is a v3 model, and yolo11n_s8_v3_s3 is around 26 ms, so I believe your test did not measure the actual latency of your own model. In my test with nc=10, the model latency is about 135 ms.

[sun-xiangyu]

tie728_s8_conv2d_h_w_unaligned_c_n_activation_loop0_no_preload_bias

Please try the latest esp-dl v3.3.3, as a bug with TIE728 has just been fixed that closely resembles this error.

[zongxiongw]

Hey @Ginosko-mia and @孙向羽，sorry to bother again.

First off, thanks for the previous advice. I've switched to the cat_detect example with esp-dl v3.3.3, upgraded esp-ppq to the latest, and re-quantized a new .espdl file. It's sitting in cat_detect/main/models/s3/ now.

But here's the weird thing that's been bugging me all day.

The official cat.jpg runs just fine — I get results like category: 0, score: 0.85 every time, inference takes about 135ms, no crashes. So the model and the project seem healthy.

However, as soon as I replace cat.jpg with my own digit images, there's zero detection output. The logs show the model loads fine and inference completes, but no results at all.

I've tried:

• Original images pulled directly from my SVHN training set
• Photos of printed digits taken with my phone
• WPS screenshots of typed digits
• A black-background white-text "5" generated with PIL

For all of these, I've made sure:

• They're real JPEGs, converted with PIL or Paint, not just renamed
• I run idf.py fullclean before every rebuild
• And switching back to the original cat.jpg always works again immediately

So I'm kinda stuck here. A few questions:

1. Is it possible my model didn't actually learn digit features at all, just memorized some background patterns? Otherwise I can't explain why it misdetects the cat as "0" but completely ignores real digit images.
2. Could there be some preprocessing step I'm missing before feeding images to the model? Like resize method, color space, or some subtle difference between my JPEGs and the official cat.jpg?
3. If you have a moment, could you take a look at my .espdl file? I can share a download link.

Thanks a lot!

espdet_pico_224_224_cat.zip

[Ginosko-mia]

When I run inference with the .espdl file you uploaded, I get the same error described in your previous issue.
Can you upload a .pt model from a deprecated version? @zongxiongw

When you replace the model or test image, please note the following:

1. Replacing the .espdl model
   Replace the file at:
   esp-dl/models/cat_detect/models/s3/espdet_pico_224_224_cat.espdl
   with your own .espdl file, but keep the filename unchanged: it must still be named espdet_pico_224_224_cat.espdl.
2. Replacing the test image
   Replace the file at:
   esp-dl/examples/cat_detect/main/cat.jpg
   with your own .jpg image, but keep the filename unchanged: it must still be named cat.jpg.

[zongxiongw]

Hi @Ginosko-mia,
Here's the
best.zip
Please let me know if you need anything else.

[Ginosko-mia]

@zongxiongw

1. First, best.pt itself was not trained well, so detection performance is poor. Here is the test image:

Here is the test result from running best.pt:

2. I am not sure how you obtained your .espdl file. The .espdl you uploaded has issues. Please refer to export.py and quantize.py under deploy for model conversion and quantization.
3. I used the best.pt you provided, used the test image as the calibration set, and after quantization the resulting .espdl runs inference on the board correctly. Here is the on-chip runtime result:

4. What you need to do now: prepare a suitable calibration set (I recommend at least 20 images), then re-export the ONNX model and quantize it again, and re-test. When quantizing, pay close attention to quantization error.

5. default_score_thr is 0.6 in cat_detect, Please change it to 0.25 for debugging.

[zongxiongw]

Hi @Ginosko-mia,

I've been debugging this issue for days and want to give you a clear summary of what I've tried and where I'm stuck.

📌 Original Problem (recap)
My first run of the full pipeline crashed right after ONNX export due to a script bug:

text
TypeError: expected str, bytes or os.PathLike object, not tuple
So the quantization step never ran.

✅ Corrected Pipeline (what I've done now)
Re-exported ONNX with the official deploy/export.py:

text
python -c "from deploy.export import Export; Export('runs/detect/train-3/weights/best.pt', 224)"
Prepared a dedicated calibration set: I created a new folder datasets/svhn_digit/calib_v2/ and manually selected at least 20 diverse SVHN digit images (different digits, backgrounds, lighting).

Ran quantization with explicit parameters:

text
python deploy/quantize.py --model "runs/detect/train-3/weights/best.onnx" --target "esp32s3" --size 224 224 --calib_data "datasets/svhn_digit/calib_v2" --espdl "espdet_pico_224_224_cat.espdl"
Copied the new .espdl into cat_detect/main/models/s3/espdet_pico_224_224_cat.espdl.

In main/app_main.cpp, I modified the code to lower the score threshold:

cpp
CatDetect *detect = new CatDetect();
detect->set_score_thr(0.25);
Every build is preceded by idf.py fullclean.

❌ Current Behavior
The official cat.jpg still works fine (outputs category: 0).

Any of my own digit images (including original SVHN training images) produce absolutely no detection output. No errors, no crashes — the inference runs, but zero results.

I also did a controlled test: I downloaded the exact test image you used in your demo, renamed it to cat.jpg, and replaced the file. Even with your own test image, there is still no detection output from my model.

🙏 Request
Could you please share the exact command and steps you used to convert my best.pt into a working .espdl? I'd like to replicate your process exactly. If possible, could you also check the newly generated .espdl file? I can provide a download link.

Thank you so much for your patience

[Ginosko-mia]

You can upload your whole project, including .pt, scripts, .onnx & the cat detect project.

What I have done:
python -m deploy.export
python -m deploy.quantize
This is the whole process. Remember to change the parameters.

export.py

if __name__ == '__main__':
    Export("/home/ginosko/best.pt", 224)

quantize.py

if __name__ == "__main__":
    quant_espdet(
        onnx_path="/home/ginosko/best.onnx",
        target="esp32s3",
        num_of_bits=8,
        device='cpu',
        batchsz=32,
        imgsz=224,
        calib_dir="deploy/cat_calib",
        espdl_model_path="best.espdl",
    )

[zongxiongw]

esp-dl-debug.zip

Hi @Ginosko-mia,

I've followed your exact steps but still can't get my digit model to work. Here's a quick summary:

What I did (exactly as you instructed):

Modified deploy/export.py and deploy/quantize.py with the parameters you specified.

Ran python -m deploy.export and python -m deploy.quantize successfully.

Used deploy/cat_calib as calibration set.

Copied the new .espdl to cat_detect/main/models/s3/espdet_pico_224_224_cat.espdl.

Lowered score threshold to 0.25 in app_main.cpp.

Even used your own test image as cat.jpg.

Current behavior:

The official cat.jpg still outputs category: 0.

Any digit image (including your demo image, SVHN training images, and PIL-generated digits) produces zero detection output. No errors, no crashes — just nothing.

My request:
I've uploaded the complete project (including best.pt, best.onnx, modified scripts, cat_detect project, and the newly generated .espdl). Could you please check what's going wrong? I suspect the model might not be loading correctly on the board.

[Ginosko-mia]

I think there's something wrong with your project layout — most likely the model file is not in the expected location. I've described how to swap in a different .espdl model in an earlier message; please refer back to that. With the .espdl you shared, I was able to obtain the following result.

[zongxiongw]

So my espdl file itself is fine. Could it be that I placed the file incorrectly in the cat_detect project? I thought it only needed to be put under the main folder.

[Ginosko-mia]

#16 (comment)

[zongxiongw]

I strictly followed all your instructions. I replaced the file espdl/models/cat_detect/models/s3/espdet_pico_224_224_cat.espdl with the .espdl file I trained and generated, and kept the original filename exactly the same. I also handled the test image in the way you specified. I really don't understand what you mean by a structural problem.

[Ginosko-mia]

After placing the model at models/cat_detect/models/s3/espdet_pico_224_224_cat.espdl, are you able to reproduce the result? The test image is the one I've provided. link

[zongxiongw]

I followed your suggestion and re-created a clean cat_detect project from scratch. The fresh project contains only the main and img folders, along with the standard configuration files. There is no models or espdl_models directory by default.

I then placed my .espdl file at the exact path you specified: models/cat_detect/models/s3/espdet_pico_224_224_cat.espdl. However, the build still fails with the following error:

text
ninja: error: '/espdet_pico_224_224_cat.espdl', needed by 'espdl_models/cat_detect.espdl', missing and no known rule to make it
I started looking deeper and noticed that the main/CMakeLists.txt file only embeds the test image, not the model file. Here is the current content of main/CMakeLists.txt:

cmake
set(src_dirs ./)
set(include_dirs ./)
set(requires cat_detect)

if (IDF_TARGET STREQUAL "esp32s3")
list(APPEND requires esp32_s3_eye_noglib
esp_lcd)
elseif (IDF_TARGET STREQUAL "esp32p4")
list(APPEND requires esp32_p4_function_ev_board_noglib
esp_lcd)
endif()

set(embed_files "cat.jpg")

idf_component_register(SRC_DIRS ${src_dirs} INCLUDE_DIRS ${include_dirs} REQUIRES ${requires} EMBED_FILES ${embed_files})
As you can see, the EMBED_FILES list only contains cat.jpg. Should I add the model file to this list? If so, what is the correct relative path for the model file that should be used in this file? Or is the model file expected to be embedded elsewhere (maybe by the cat_detect component itself)?

I would appreciate any clarification on what needs to be changed in this CMakeLists.txt (or any other file) so that the build succeeds and my custom digit model is actually used instead of the default cat model.

Thank you again for your help.