🐛 Bug: Alert is persisted and audited as triggered, but matching event workflow is not executed

[haonanan]

Summary

A single interval workflow whose latest workflowexecution.status is stuck at in_progress
causes the scheduler to spam Workflow <id> is already running by someone else and, as a side
effect, event-driven (alert) workflows stop being dispatched even though alerts continue to
be ingested into the alert table.

In our case ~57,000 spam log lines were produced in ~32 hours and at least 77 alert events
(across multiple workflows / fingerprints) were ingested but never produced a corresponding
workflowexecution row.

Environment

• Keep version: v0.51.0
• Deployment: docker-compose (single backend container)
• DB: PostgreSQL 15
• Redis: enabled (REDIS=true)
• Auth: oauth2-proxy / keycloak

Repro

1. Create or already-have an interval workflow (interval > 0, e.g. a daily report workflow).
2. Cause its latest workflowexecution to be left at status='in_progress' (e.g. backend
   container crash / OOM kill / network blip during execution, before INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT
   elapses, or any path that leaves the row not finished and not timed out yet — in our case
   we don't yet know the exact crash trigger, but the row simply stays in_progress forever
   because nothing updates it).
3. Observe backend logs:

   <ts> DEBUG keep.api.core.db: Workflow <id> is already running by someone else
   <ts> DEBUG keep.api.core.db: Workflow <id> is already running by someone else
   ... (repeats every scheduler iteration, ~1/s)
   

4. Send alert events to other (event-triggered) workflows. Observe that:
   • alert table receives the rows.
   • No corresponding workflowexecution row is created for the event workflows that
     would otherwise have matched.
5. UPDATE workflow SET is_disabled = true WHERE id = '<stuck-id>'; — spam stops within ~1
   minute and event dispatch resumes.

Evidence from our incident

-- 77 alert events with no workflowexecution after them, across ~3 fingerprints
SELECT a.fingerprint, COUNT(*) AS alerts_in,
       SUM(CASE WHEN we.id IS NULL THEN 1 ELSE 0 END) AS missed
  FROM alert a
  LEFT JOIN workflowexecution we
    ON we.triggered_by LIKE 'type:alert%' AND we.started >= a.timestamp
                                          AND we.started <= a.timestamp + interval '60s'
 WHERE a.timestamp >= '2026-05-25 18:00:00'
 GROUP BY 1;

(numbers from our DB; redacted)

Root-cause analysis

The scheduler main loop is strictly serial:

https://github.com/keephq/keep/blob//keep/workflowmanager/workflowscheduler.py#L648

while not self._stop:
    self._handle_interval_workflows()
    self._handle_event_workflows()
    time.sleep(1)

_handle_interval_workflows() calls get_workflows_that_should_run():

https://github.com/keephq/keep/blob//keep/api/core/db.py#L319

For every interval workflow it does:

• get_last_completed_execution(...)
• if there's an ongoing_execution.status == 'in_progress' and it hasn't passed
  INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT yet, it just logs is already running by someone else
  and continues.

If INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT is long (or never reached because of clock/state),
the row stays in_progress forever and every scheduler iteration re-runs the same N SQL
statements per interval workflow. Under our connection pool / DB latency, this stretches a
single iteration enough that _handle_event_workflows() falls far behind, and (we suspect)
the workflows_to_run queue contention or DB pool starvation causes event workflow dispatch
to silently drop events.

What we'd like to see

1. Decouple interval and event scheduling, e.g. run them on independent threads / tasks,
   so a slow/stuck interval workflow cannot starve event dispatch.
2. Self-heal stuck in_progress rows more aggressively (e.g. shorter default
   INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT, or a watchdog that finishes orphans on backend
   startup based on container restart time).
3. Rate-limit the is already running log (one per workflow per N minutes is enough).
4. Optional: expose a /healthz / metric that surfaces "stuck interval workflows count" so
   ops can alert on it.

Workaround

UPDATE workflow         SET is_disabled = true                        WHERE id = '<stuck-id>';
UPDATE workflowexecution SET is_running = 0, status='error',
                             error='manual cleanup of stuck execution'
 WHERE workflow_id = '<stuck-id>' AND is_running = 1;

Spam stops within ~1 minute, event-workflow dispatch resumes immediately.

Additional notes

• arq worker / process_event_task itself completes (we see it writing to alert and
  alertdeduplicationrule); the missing piece is the workflow scheduler picking the alert
  up from the in-memory workflows_to_run queue.
• Default deduplication rule fires normally during the incident, ruling out dedup as the
  cause of the missing dispatches.

Happy to share more logs / SQL output if helpful.

[dosubot]

Thanks for the detailed bug report and root-cause analysis — this is a well-documented and valid issue.

I can confirm from the codebase that the scheduler loop is indeed strictly sequential [1]:

self._handle_interval_workflows()
self._handle_event_workflows()

And that INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT is hardcoded to 60 minutes [2], which means a stuck workflow will spam logs at ~1Hz for up to an hour before self-healing kicks in.

A few additional notes that may help contextualize the issue:

1. There is a universal timeout handler (_timeout_workflows()) that runs every ~100 scheduler iterations (~100 seconds) and covers all workflow types [3]. However, this only helps if the execution's started timestamp + timeout has elapsed — if the 60-minute threshold hasn't passed, it won't clean up the stuck row either.

2. Connection pool exhaustion may be compounding the problem. PR fix: close leaked SQLAlchemy sessions in process_watcher_task #5611 [4] identified that SQLAlchemy session leaks in process_watcher_task (maintenance window operations) can exhaust the connection pool in ~30 minutes. If your deployment is affected by this, the repeated IntegrityError → query → status check cycle for the stuck workflow would hang on pool acquisition, making _handle_interval_workflows() take even longer and further starving event dispatch. Worth checking if you see "idle in transaction" connections accumulating.

3. The locking mechanism uses a database unique constraint on (workflow_id, execution_number, is_running, timeslot) [5]. Each scheduler iteration for a stuck workflow triggers: an attempted INSERT (fails with IntegrityError), a SELECT to fetch the ongoing execution, and a status/runtime check — so under connection pool pressure, this multiplies the blocking effect.

Your proposed fixes all make sense architecturally. The decoupling of interval and event scheduling (suggestion #1) would be the most impactful, since the sequential model is the fundamental design choice that allows one subsystem to block the other. A shorter or configurable INTERVAL_WORKFLOWS_RELAUNCH_TIMEOUT (suggestion #2) would reduce the window of impact significantly.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[haonanan]

Update / correction after deeper DB tracing:

After additional investigation, I want to soften the original root-cause claim.

The stuck interval workflow and the Workflow <id> is already running by someone else
log spam were real, and disabling that workflow stopped the spam. However, I can no longer
prove that this stuck interval workflow was the direct cause of the missed event workflow
dispatches.

What we can prove from persistent DB state is narrower:

• The alert row was inserted into alert.
• alertaudit recorded alert was triggered.
• The alert payload matched the workflow trigger filters:
  • status = firing
  • labels.sub_project = samo
  • labels.game_project = samo
  • and the step condition for the QA route:
    • labels.environment = qa
    • labels.alert_category = business
• But no workflowexecution row was created.
• No workflowtoalertexecution row was created for that alert event_id.

Example from our DB:

-- Suspicious alert event
SELECT id, timestamp, fingerprint,
       event->>'status' AS status,
       event->'labels'->>'environment' AS env,
       event->'labels'->>'alert_category' AS category,
       event->>'isPartialDuplicate' AS partial_dup,
       event->>'firingCounter' AS firing_counter
  FROM alert
 WHERE id = '<alert-id>';

Result:

timestamp:       2026-05-25 14:35:16 UTC
fingerprint:     xxxxxxxx
partial_dup:     true
firingCounter:   94

The alert was audited:

SELECT timestamp, action, user_id, description
  FROM alertaudit
 WHERE fingerprint = '<fingerprint>'
   AND timestamp BETWEEN '2026-05-25 14:34:00' AND '2026-05-25 14:36:00';
Result:

2026-05-25 14:35:16 | alert was triggered | system | Alert received from provider with status firing

But no workflow execution link exists:

SELECT *
FROM workflowtoalertexecution
WHERE event_id = '';

Result:

0

And there were no workflow executions around that exact time window:

```sql
SELECT *
  FROM workflowexecution
 WHERE started BETWEEN '2026-05-25 14:33:00' AND '2026-05-25 14:38:00';

Result:

0 rows

So the strongest confirmed statement is:

Keep persisted and audited the alert as triggered, but no workflow execution/link row was
created for an alert that matched the workflow filters.

This points to an intermittent gap somewhere between alert persistence/audit and workflow
execution creation. The stuck interval workflow may still be related, but I cannot prove
that causal link from the remaining DB evidence alone, because the backend logs for the
incident window were lost after container recreation.

A more accurate issue title might be:

Intermittent gap between alert persistence/audit and workflow execution creation

or:

Alert is persisted and audited as triggered, but matching event workflow is not executed

The original scheduler-starvation hypothesis is still possible, but should be treated as a
hypothesis rather than a confirmed root cause.

[dosubot]

Thanks for the honest correction — narrowing this to "alert persisted and audited, but no workflow execution created" is a much more actionable framing.

I dug into the code path from alert ingestion to workflow dispatch, and there are a few candidate explanations for the gap you're seeing. The good news: isPartialDuplicate: true is not a factor — only full duplicates are filtered before workflow evaluation [1].

Here are the most likely silent failure paths:

1. Pusher client null check causing early return (most suspicious)

In __handle_formatted_events(), there's a check:

pusher_client = get_pusher_client() if notify_client else False
if not pusher_client:
    return

This runs after the alert is already saved and audited, but before workflow_manager.insert_events() is called [2]. If get_pusher_client() returns None (missing PUSHER_APP_ID, PUSHER_APP_KEY, PUSHER_APP_SECRET, or PUSHER_DISABLED=true), workflow dispatch is completely bypassed. Since you're on docker-compose with websockets likely via Pusher, an intermittent connection issue or misconfiguration could explain the gap.

2. Silent exception in insert_events()

The workflow insertion is wrapped in a bare try/except [2]:

try:
    workflow_manager.insert_events(tenant_id, enriched_formatted_events)
except Exception:
    logger.exception("Failed to run workflows based on alerts")

If this throws, it's logged but not re-raised — alerts stay persisted, workflows silently don't fire.

3. CEL expression evaluation failure

Inside WorkflowManager.insert_events(), CEL filter expressions are evaluated per-alert. If evaluation throws (e.g., missing has() guard on a label, casing mismatch), the alert is silently skipped for that workflow [3].

Diagnostic suggestions (since you lost the logs from the incident window):

• Verify your Pusher env vars (PUSHER_APP_ID, PUSHER_APP_KEY, PUSHER_APP_SECRET) are set and stable. If you're not using Pusher at all and have PUSHER_DISABLED=true, that early-return path would explain the behavior.
• On next occurrence, search backend logs for "Failed to run workflows based on alerts" or "Error evaluating CEL for event in insert_events".
• Check whether KEEP_AUDIT_EVENTS_ENABLED is set — if disabled, workflowtoalertexecution records won't be created even for successful executions (though workflowexecution rows should still exist).

The Pusher dependency path (#1) is architecturally concerning because it couples real-time UI notifications to workflow execution correctness. If that turns out to be the cause, it would be worth filing separately as a design issue.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.