Skip to content

backend: add alerting for stuck async operations #5952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Jul 11, 2026
+1,319 −0
Merged

backend: add alerting for stuck async operations #5952

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
361 changes: 361 additions & 0 deletions backend/alerts/backend-async-operations-prometheusRule.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,361 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
app.kubernetes.io/name: kube-prometheus
app.kubernetes.io/part-of: kube-prometheus
prometheus: k8s
role: alert-rules
name: backend-async-operations-rules
namespace: monitoring
spec:
groups:
# ========================================
# Backend Async Operation Stuck Alerts
# ========================================
# These alerts fire when an async operation has been running longer than
# the corresponding e2e test timeout. Thresholds are anchored to the
# canonical timeout constants in test/util/framework/constants.go.
Comment on lines +16 to +18
#
# Metric: backend_resource_operation_start_time_seconds (total wall-clock
# time since the operation was created). Combined with phase_info to
# restrict to operations still in non-terminal phases.
#
# No `for` clause: the phase filter already ensures we only alert on
# in-flight operations. Once an operation reaches succeeded/failed/canceled,
# the phase no longer matches and the alert resolves immediately.
#
# All expressions use max by(...) to deduplicate across Prometheus
# replicas/shards, preventing duplicate alerts for the same operation.
- name: backend-async-operations
rules:
# ----------------------------------------
# Cluster operations
# ----------------------------------------
# Cluster create: e2e timeout = 20 minutes (ClusterCreationTimeout)
- alert: BackendAsyncOperationClusterCreateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="create"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="create",
phase=~"accepted|provisioning"
}
) == 1
) > 1200
Comment thread
stevekuznetsov marked this conversation as resolved.
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Cluster create operation stuck'
description: 'Cluster create operation for {{ $labels.resource_id }} has been running for over 20 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: ClusterCreationTimeout).'
runbook_url: TBD
# Cluster update: e2e timeout = 45 minutes (HCPClusterVersionUpgradeTimeout)
# This covers both property updates and version upgrades, since both
# use operation_type=update. The higher timeout accommodates upgrades.
- alert: BackendAsyncOperationClusterUpdateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="update"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="update",
phase=~"accepted|updating"
}
) == 1
) > 2700
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Cluster update operation stuck'
description: 'Cluster update operation for {{ $labels.resource_id }} has been running for over 45 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: HCPClusterVersionUpgradeTimeout).'
runbook_url: TBD
# Cluster delete: e2e timeout = 25 minutes (HCPClusterDeletionTimeout)
# Delete operations start directly in "deleting" phase, skipping "accepted".
- alert: BackendAsyncOperationClusterDeleteStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="delete"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="delete",
phase="deleting"
}
) == 1
) > 1500
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Cluster delete operation stuck'
description: 'Cluster delete operation for {{ $labels.resource_id }} has been running for over 25 minutes and is currently in deleting phase (e2e timeout: HCPClusterDeletionTimeout).'
runbook_url: TBD
# ----------------------------------------
# Credential operations
# ----------------------------------------
# Request credential: e2e timeout = 10 minutes (GetAdminRESTConfigTimeout)
- alert: BackendAsyncOperationCredentialRequestStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="requestcredential"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="requestcredential",
phase=~"accepted|provisioning"
}
) == 1
) > 600
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Credential request operation stuck'
description: 'Credential request operation for {{ $labels.resource_id }} has been running for over 10 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: GetAdminRESTConfigTimeout).'
runbook_url: TBD
# Revoke credentials: e2e timeout = 15 minutes (ExternalAuthDeletionTimeout used as proxy)
- alert: BackendAsyncOperationCredentialRevokeStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="revokecredentials"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters",
operation_type="revokecredentials",
phase=~"accepted|deleting"
}
) == 1
) > 900
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Credential revoke operation stuck'
description: 'Credential revoke operation for {{ $labels.resource_id }} has been running for over 15 minutes and is currently in {{ $labels.phase }} phase.'
runbook_url: TBD
# ----------------------------------------
# Node pool operations
# ----------------------------------------
# Node pool create: e2e timeout = 20 minutes (NodePoolCreationTimeout)
- alert: BackendAsyncOperationNodePoolCreateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="create"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="create",
phase=~"accepted|provisioning"
}
) == 1
) > 1200
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Node pool create operation stuck'
description: 'Node pool create operation for {{ $labels.resource_id }} has been running for over 20 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: NodePoolCreationTimeout).'
runbook_url: TBD
# Node pool update: e2e timeout = 45 minutes (NodePoolVersionUpgradeTimeout)

@SudoBrendan SudoBrendan Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You pulled this straight from the test case, but I wonder if that test itself is wrong and conflating various kinds of "update", maybe it was from the before-times when we hadn't decided to fire-and-forget on OCP version upgrades? As I understand from a backend Operation perspective, the actual LRO should go basically from Accepted -> Succeeded/Failed immediately on an OCP upgrade, right? We should maybe consider a shorter timeout, so that when customers bump their autoscale max from 8 -> 9 and that gets locked it doesn't take 45m to fire (here, and in the test, honestly)

However... while investigating this - I did discover that there are some LROs that we could expect to go longer than 45m or even an hour - internal discussion here.

@stevekuznetsov stevekuznetsov Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, if/when we change the e2e, we can change this - and you're 100% right that there are possible updates that could take a long time. If and unless we update the metric to classify the 'type' of update LRO, we can't subdivide. Time will tell if it is meaningful or not.

# Covers both property updates and version upgrades.
- alert: BackendAsyncOperationNodePoolUpdateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="update"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="update",
phase=~"accepted|updating"
}
) == 1
) > 2700
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Node pool update operation stuck'
description: 'Node pool update operation for {{ $labels.resource_id }} has been running for over 45 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: NodePoolVersionUpgradeTimeout).'
runbook_url: TBD
# Node pool delete: e2e timeout = 25 minutes (NodePoolDeletionTimeout)
- alert: BackendAsyncOperationNodePoolDeleteStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="delete"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/nodepools",
operation_type="delete",
phase="deleting"
}
) == 1
) > 1500
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Node pool delete operation stuck'
description: 'Node pool delete operation for {{ $labels.resource_id }} has been running for over 25 minutes and is currently in deleting phase (e2e timeout: NodePoolDeletionTimeout).'
runbook_url: TBD
# ----------------------------------------
# External auth operations
# ----------------------------------------
# External auth create: e2e timeout = 15 minutes (ExternalAuthCreationTimeout)
# Includes the awaitingsecret phase, which is specific to external auth.
- alert: BackendAsyncOperationExternalAuthCreateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="create"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="create",
phase=~"accepted|awaitingsecret|provisioning"
}
) == 1
) > 900
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'External auth create operation stuck'
description: 'External auth create operation for {{ $labels.resource_id }} has been running for over 15 minutes and is currently in {{ $labels.phase }} phase (e2e timeout: ExternalAuthCreationTimeout).'
runbook_url: TBD
# External auth update: no explicit e2e constant, using 10 minutes as
# a reasonable default (no VM provisioning involved).
- alert: BackendAsyncOperationExternalAuthUpdateStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="update"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="update",
phase=~"accepted|updating"
}
) == 1
) > 600
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'External auth update operation stuck'
description: 'External auth update operation for {{ $labels.resource_id }} has been running for over 10 minutes and is currently in {{ $labels.phase }} phase.'
runbook_url: TBD
# External auth delete: e2e timeout = 15 minutes (ExternalAuthDeletionTimeout)
- alert: BackendAsyncOperationExternalAuthDeleteStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="delete"
}
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
resource_type="microsoft.redhatopenshift/hcpopenshiftclusters/externalauths",
operation_type="delete",
phase="deleting"
}
) == 1
) > 900
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'External auth delete operation stuck'
description: 'External auth delete operation for {{ $labels.resource_id }} has been running for over 15 minutes and is currently in deleting phase (e2e timeout: ExternalAuthDeletionTimeout).'
runbook_url: TBD
# ----------------------------------------
# Backstop: any async operation not covered above
# ----------------------------------------
# Catches any future operation types that don't have explicit alerts.
# Threshold (60 min) is higher than the longest specific alert (45 min for upgrades).
# Double-fires with specific alerts correlate to the same IcM incident via shared correlationId.
- alert: BackendAsyncOperationStuck
expr: |
(
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
time() - backend_resource_operation_start_time_seconds
)
and
max by (resource_id, subscription_id, resource_type, operation_type, phase, cluster) (
backend_resource_operation_phase_info{
phase=~"accepted|provisioning|updating|deleting|awaitingsecret"
}
) == 1
) > 3600
labels:
severity: "3"
annotations:
correlationId: 'BackendAsyncOperationStuck/{{ $labels.cluster }}/{{ $labels.resource_id }}'
summary: 'Async operation stuck'
description: 'Async {{ $labels.operation_type }} operation for {{ $labels.resource_id }} ({{ $labels.resource_type }}) has been running for over 60 minutes and is currently in {{ $labels.phase }} phase.'
runbook_url: TBD
Loading