Skip to content

feat(automation): run timeout, retry/backoff policy, and force-stop action - #636

Open
cagdasyurekli wants to merge 1 commit into
andrewyng:mainfrom
cagdasyurekli:feat/issue-621-scheduler-timeout-retries
Open

feat(automation): run timeout, retry/backoff policy, and force-stop action#636
cagdasyurekli wants to merge 1 commit into
andrewyng:mainfrom
cagdasyurekli:feat/issue-621-scheduler-timeout-retries

Conversation

@cagdasyurekli

Copy link
Copy Markdown

Resolves #621

Summary

Implements run timeouts, exponential backoff retries for failed automation runs, and an explicit force-stop action so that hung tasks cannot silently hold the scheduler overlap guard indefinitely.

Key Changes

  1. Per-task & Default Run Timeout (coworker/automation/models.py, coworker/automation/scheduler.py):
    • Added timeout_seconds: Optional[float] = 900.0 (default 15 minutes) to ScheduledTask.
    • Scheduler wraps run execution in asyncio.wait_for(...). When the timeout fires:
      • Run status is marked timed_out with descriptive error.
      • Scheduler unconditionally releases the overlap guard in finally:.
      • An on_timeout notification is parked in the user's InboxStore and broadcast via WebSocket toast events.
  2. Error Retry & Exponential Backoff (coworker/automation/scheduler.py, coworker/automation/store.py):
    • Added max_retries: int = 0 and retry_backoff_seconds: float = 60.0 to ScheduledTask.
    • If a run fails with status == "error" and attempts remain (fresh.retry_count < fresh.max_retries):
      • Calculates backoff retry_backoff_seconds * (2 ** retry_count).
      • Advances retry_count += 1 and schedules next run at now + backoff without clobbering cron schedule.
      • Run trigger is marked as "retry".
    • Successful runs or user cancellations reset retry_count = 0 and return to the primary schedule.
  3. Explicit Force-Stop (coworker/automation/scheduler.py, coworker/server/manager.py, coworker/server/app.py, surfaces/gui/):
    • Scheduler.force_stop(task_id) immediately discards _running_ids and cancels the in-flight asyncio.Task.
    • manager.force_stop_automation(task_id) halts running task engines and persists a cancelled run record.
    • Added POST /v1/automations/{task_id}/stop endpoint in FastAPI.
    • Added stopAutomation API function and UI "Force Stop" buttons in ScheduledView.tsx.
  4. Comprehensive Unit Tests (tests/test_automation.py):
    • Verified timeout triggers, timeout notification callback, and overlap guard release allowing subsequent executions.
    • Verified exponential retry progression and retry exhaustion back into normal cron scheduling.
    • Verified force stop cancellation and POST /v1/automations/{task_id}/stop endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Scheduled-task run timeout + retry/backoff policy

1 participant