Skip to content

Catch, log, and handle exception in pipeline#7017

Merged
A-Artemis merged 2 commits into
developfrom
aurelien/catch-log-and-handle-pipeline-errors
Jul 15, 2026
Merged

Catch, log, and handle exception in pipeline#7017
A-Artemis merged 2 commits into
developfrom
aurelien/catch-log-and-handle-pipeline-errors

Conversation

@A-Artemis

@A-Artemis A-Artemis commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Enhance resilience of event dispatching and stream loading by handling errors gracefully

Checklist

  • The PR title and description are clear and descriptive
  • I have manually tested the changes
  • All changes are covered by automated tests
  • All related issues are linked to this PR (if applicable)
  • Documentation has been updated (if applicable)

@A-Artemis A-Artemis self-assigned this Jul 13, 2026
Copilot AI review requested due to automatic review settings July 13, 2026 12:11
@A-Artemis
A-Artemis requested a review from a team as a code owner July 13, 2026 12:11
@github-actions github-actions Bot added TEST Any changes in tests Geti Backend Issues related to the Geti application server labels Jul 13, 2026
@A-Artemis A-Artemis linked an issue Jul 13, 2026 that may be closed by this pull request
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

⏱️ Backend import time — app.main

Accelerator Cumulative (s) Self (s)
cpu 4.546 0.004
xpu 4.550 0.003
cuda 4.627 0.004

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves backend pipeline resilience by catching and handling exceptions in key worker loops (event dispatching, stream loading, inference callback processing, and inference-server TTL monitoring) so that transient or isolated failures don’t crash long-running workers.

Changes:

  • Add retry/backoff + consecutive-error thresholding for frame acquisition and reopen attempts in StreamLoader.
  • Make event emission and dispatching more fault-tolerant by isolating failing handlers/destinations and keeping loops alive.
  • Extend unit tests to cover “survives failure” behavior across stream loading, dispatching, event bus, and inference-server monitoring.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
application/backend/app/workers/stream_loading.py Adds consecutive-error handling, exponential backoff, and stream lifecycle guards for stream acquisition.
application/backend/app/workers/dispatching.py Makes sink initialization/connection and dispatch loop more resilient to exceptions; isolates per-destination failures.
application/backend/app/services/event/base.py Prevents one failing subscriber from breaking event emission to other subscribers.
application/backend/app/workers/inference.py Wraps inference completion callback to prevent exceptions from wedging the async inference queue.
application/backend/app/workers/inference_server_monitor.py Prevents TTL monitor thread from dying when model unload fails.
application/backend/tests/unit/workers/test_stream_loading.py Updates/extends tests for consecutive frame error thresholding and transient failure recovery.
application/backend/tests/unit/workers/test_dispatching.py Adds resilience tests for sink connect/build failures and data collector failures.
application/backend/tests/unit/services/event/test_event_bus.py Adds test ensuring emit_event continues after a failing handler.
application/backend/tests/unit/workers/test_inference_server_monitor.py Adds test ensuring unload failure doesn’t kill the monitor loop.

Comment thread application/backend/app/workers/dispatching.py
Comment thread application/backend/app/workers/stream_loading.py Outdated
Comment thread application/backend/app/workers/inference_server_monitor.py Outdated
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

📊 Test coverage report

Metric Coverage
Lines 60.3%
Functions 57.0%
Branches 52.1%
Statements 59.9%

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown

🐳 Docker image sizes

Device Size
xpu 10801.8 MB (10.55 GB)
cpu 4013.2 MB (3.92 GB)
cuda 10208.4 MB (9.97 GB)

Comment thread application/backend/app/workers/inference_server_monitor.py

@leoll2 leoll2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. This very defensive exception handling may seem excessive at first glance, but I think it's necessary in these delicate yet critical pipeline components. I appreciate the very descriptive inline comments, I find them really helpful to understand what every protection mechanism is intended for.

@A-Artemis
A-Artemis added this pull request to the merge queue Jul 15, 2026
Merged via the queue into develop with commit 23460b8 Jul 15, 2026
39 checks passed
@A-Artemis
A-Artemis deleted the aurelien/catch-log-and-handle-pipeline-errors branch July 15, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Geti Backend Issues related to the Geti application server TEST Any changes in tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Catch, log and handle errors in pipeline flow

3 participants