Catch, log, and handle exception in pipeline#7017
Conversation
…g errors gracefully
⏱️ Backend import time —
|
| Accelerator | Cumulative (s) | Self (s) |
|---|---|---|
cpu |
4.546 | 0.004 |
xpu |
4.550 | 0.003 |
cuda |
4.627 | 0.004 |
There was a problem hiding this comment.
Pull request overview
This PR improves backend pipeline resilience by catching and handling exceptions in key worker loops (event dispatching, stream loading, inference callback processing, and inference-server TTL monitoring) so that transient or isolated failures don’t crash long-running workers.
Changes:
- Add retry/backoff + consecutive-error thresholding for frame acquisition and reopen attempts in
StreamLoader. - Make event emission and dispatching more fault-tolerant by isolating failing handlers/destinations and keeping loops alive.
- Extend unit tests to cover “survives failure” behavior across stream loading, dispatching, event bus, and inference-server monitoring.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| application/backend/app/workers/stream_loading.py | Adds consecutive-error handling, exponential backoff, and stream lifecycle guards for stream acquisition. |
| application/backend/app/workers/dispatching.py | Makes sink initialization/connection and dispatch loop more resilient to exceptions; isolates per-destination failures. |
| application/backend/app/services/event/base.py | Prevents one failing subscriber from breaking event emission to other subscribers. |
| application/backend/app/workers/inference.py | Wraps inference completion callback to prevent exceptions from wedging the async inference queue. |
| application/backend/app/workers/inference_server_monitor.py | Prevents TTL monitor thread from dying when model unload fails. |
| application/backend/tests/unit/workers/test_stream_loading.py | Updates/extends tests for consecutive frame error thresholding and transient failure recovery. |
| application/backend/tests/unit/workers/test_dispatching.py | Adds resilience tests for sink connect/build failures and data collector failures. |
| application/backend/tests/unit/services/event/test_event_bus.py | Adds test ensuring emit_event continues after a failing handler. |
| application/backend/tests/unit/workers/test_inference_server_monitor.py | Adds test ensuring unload failure doesn’t kill the monitor loop. |
📊 Test coverage report
|
🐳 Docker image sizes
|
leoll2
left a comment
There was a problem hiding this comment.
Looks good to me. This very defensive exception handling may seem excessive at first glance, but I think it's necessary in these delicate yet critical pipeline components. I appreciate the very descriptive inline comments, I find them really helpful to understand what every protection mechanism is intended for.
Summary
Enhance resilience of event dispatching and stream loading by handling errors gracefully
Checklist