Process execution failed: TensorRT Engine build timed out. And Attempting to load inswapper_128.fp16.onnx... taking ages

[Elgokoo]

Hi I have an issue with V3 that takes ages to build , Im using the portable version

[INFO] Video framerate is Constant (CFR). Audio sync should be perfect.
[INFO] TensorRT load detected for Inswapper128. Running isolated probe...
[INFO] Probe attempt 1 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 1 failed with exit code -15.
[INFO] Retrying in 2 seconds...
[INFO] Probe attempt 2 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...

[Elgokoo]

[ERROR] Isolated probe failed for Inswapper128.
[ERROR] The model will not be loaded. This is likely a fatal TensorRT/CUDA error.
Traceback (most recent call last):
File "D:\VisoMaster-Fusion-v3\VisoMaster-Fusion\app\processors\models_processor.py", line 691, in load_model
raise RuntimeError(
RuntimeError: [ERROR] ONNX/TensorRT probe process failed after 3 attempts. Last exit code: -15
[ERROR] Emap could not be loaded for latent calculation.
[INFO] TensorRT load detected for Inswapper128. Running isolated probe...
[INFO] Probe attempt 1 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 1 failed with exit code -15.
[INFO] Retrying in 2 seconds...
[INFO] Probe attempt 2 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 2 failed with exit code -15.
[INFO] Retrying in 2 seconds...
[INFO] Probe attempt 3 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 3 failed with exit code -15.
[ERROR] Isolated probe failed for Inswapper128.
[ERROR] The model will not be loaded. This is likely a fatal TensorRT/CUDA error.
Traceback (most recent call last):
File "D:\VisoMaster-Fusion-v3\VisoMaster-Fusion\app\processors\models_processor.py", line 691, in load_model
raise RuntimeError(
RuntimeError: [ERROR] ONNX/TensorRT probe process failed after 3 attempts. Last exit code: -15
[ERROR] Emap could not be loaded for latent calculation.
[ERROR] calc_inswapper_latent returned None (emap unavailable). Skipping swap.

[Elricfae]

Did you get error during installation ? Have you updated your nvidia drivers ? It seems the engine creation fails on timeout... Best is to check or redo the venv and update your drivers. What system and graphic card do you use ?

[Elricfae]

The next update will ensure corrupted engine files during creation will be deleted so it won't crash the app and allows new retry without issue

[Elgokoo]

Thanks for your feedback but still the loading takes Ages

Existing installation detected. Launching VisoMaster Fusion Launcher...
[Launcher] Starting VisoMaster Fusion Launcher...
[Launcher] Checking for updates on branch 'main'...
[Launcher] Repository up to date (HEAD=2131b69)
[Launcher] Update status: up_to_date (checked just now)
[Launcher] Launcher ready.
[Launcher] Launching VisoMaster Fusion...
[INFO] Clearing GPU Memory: Unloading all models...
[INFO] Running garbage collection and clearing CUDA cache.
[INFO] GPU Memory Cleared.
[INFO] Max Threads set as 6. Will be applied on next run.
[INFO] Max Threads set as 4. Will be applied on next run.
[INFO] Max Threads set as 1. Will be applied on next run.
[INFO] Max Threads set as 4. Will be applied on next run.
[INFO] Max Threads set as 2. Will be applied on next run.
[INFO] Max Threads set as 1. Will be applied on next run.
[INFO] Max Threads set as 2. Will be applied on next run.
[INFO] FP16 Acceleration ENABLED for RetinaFace
[INFO] Loading model: RetinaFace with provider: TensorRT
[INFO] FP16 Acceleration ENABLED for Inswapper128ArcFace
[INFO] Loading model: Inswapper128ArcFace with provider: TensorRT
[INFO] Video framerate is Constant (CFR). Audio sync should be perfect.
[INFO] TensorRT load detected for Inswapper128. Running isolated probe...
[INFO] Probe attempt 1 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[INFO] Cleaning up corrupted TensorRT cache for Inswapper128 due to timeout...
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 1 failed with exit code -15.
[INFO] Cleaning up potentially corrupted TensorRT cache for Inswapper128...
[INFO] Retrying in 2 seconds...
[INFO] Probe attempt 2 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[INFO] Cleaning up corrupted TensorRT cache for Inswapper128 due to timeout...
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 2 failed with exit code -15.
[INFO] Cleaning up potentially corrupted TensorRT cache for Inswapper128...
[INFO] Retrying in 2 seconds...
[INFO] Probe attempt 3 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...
[ERROR] Probe process for Inswapper128 timed out! Terminating.
[INFO] Cleaning up corrupted TensorRT cache for Inswapper128 due to timeout...
[ERROR] Process execution failed: TensorRT Engine build timed out.
[WARN] Probe attempt 3 failed with exit code -15.
[INFO] Cleaning up potentially corrupted TensorRT cache for Inswapper128...
[ERROR] Isolated probe failed for Inswapper128.
[ERROR] The model will not be loaded. This is likely a fatal TensorRT/CUDA error.
Traceback (most recent call last):
File "D:\VisoMaster-Fusion-3-4\VisoMaster-Fusion\app\processors\models_processor.py", line 837, in load_model
raise RuntimeError(
RuntimeError: [ERROR] ONNX/TensorRT probe process failed after 3 attempts. Last exit code: -15
[ERROR] Emap could not be loaded for latent calculation.
[INFO] TensorRT load detected for Inswapper128. Running isolated probe...
[INFO] Probe attempt 1 of 3 for Inswapper128...
[ONNX Prober]: Attempting to load inswapper_128.fp16.onnx...

[Elgokoo]

Did you get error during installation ? Have you updated your nvidia drivers ? It seems the engine creation fails on timeout... Best is to check or redo the venv and update your drivers. What system and graphic card do you use ?

I have installed the last version 3.4.2 , portable and manual , redone everything

FYI

I have a Dual GPu : In laptop RTX 3070 and egpu rtx 5060 ti