feat: add cluster upgrade partial and complete SLO observability by machi1990 · Pull Request #5752 · Azure/ARO-HCP

machi1990 · 2026-06-23T13:15:44Z

What

Add cluster version selection-to-active SLO recording rules, alerts & slo dashboard

Why

feat: add UJ cluster upgrade partial and complete SLO observability

Add an alert that fires when an upgrade takes more than 20 minutes to reach partial
Add an alert that fires when an upgrade takes more than 30 minutes to complete
Add dashaboards for the SLOs

Testing

Special notes for your reviewer

Fleetwise version dashaboards will follow afterwards. (https://redhat.atlassian.net/browse/ARO-27927)
Runbook not yet there but they'll be created (https://redhat.atlassian.net/browse/ARO-22192); only the redirection from the link being mentioned is here and it points to what will be the name of the runbook as done in the corresponding WIP PR
I intend to keep the alert as info and over time graduate them to the corresponding severity based on their tier as par https://github.com/openshift-online/architecture/pull/79

PR Checklist

openshift-ci · 2026-06-23T13:15:54Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

JameelB · 2026-06-24T15:31:17Z

+      }
+      {
+        record: 'hosted_control_plane_version:selection_to_partial:desired_pending_seconds'
+        expression: '( time() - hosted_control_plane_version:selection_to_partial:state_first_seen_timestamp{state="desired"} ) and on(cluster, resource_id, subscription_id, cluster_uuid, version) ( backend_cluster_version_info{state="desired"} == 1 )'


Is the goal of the second part of this query to ensure we only capture metrics with the desired state to see how long it's been in that state (ignores when it moves to partial/completed).

Looking at the way the metrics are emitted (PR), it seems to re-emit every sync and there wouldn't be a recording rule hosted_control_plane_version:selection_to_partial:state_first_seen_timestamp{state="desired"} for it anymore cause backend_cluster_version_info == 1 for that state would not exist? If my understanding there is correct, is the second half of the query really needed?

backend_cluster_version_info{state="desired"} == 1

The intent of this was to capture "the given desired version is stuck on desired".
To only show up upgrades that are currently stuck as opposed to the version was ever stuck on desired.

Though I am wondering what's the best approach between the two differences; should we interest ourselves with the current stuck or current and any previous stuck?

swiencki · 2026-06-24T23:17:52Z

+  # Multi-window burn-rate alerts for cluster upgrade version selection-to-partial SLO. Sev 4 (info).
+  - name: rp-uj-cluster-upgrade-version-selection-to-partial-monitor-rules
+    rules:
+    - alert: UJClusterUpgradeVersionSelectionToPartial5m1h


Tier suffix should be long-window-first (1h5m, 6h30m, 3d6h) per the convention in the KAS and NodePool alerts.

swiencki · 2026-06-24T23:17:52Z

+      annotations:
+        summary: "{{ $labels.cluster }}: Cluster upgrade version selection-to-partial error budget burning rapidly (>14.4x)"
+        description: "More than 14.4x of the 99.95% cluster upgrade version selection-to-partial error budget is being consumed over 5m and 1h windows. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Investigate backend_cluster_version_info on affected clusters."
+        runbook_url: TBD


RP lane alerts need a resolvable aka.ms/arohcp-runbook-... short-link before merge, even if the runbook is still high level.

machi1990 · 2026-06-25T17:38:07Z

I am unsure if this is the correct lane to use. https://github.com/swiencki/architecture/blob/16fbc2fb1ffa04b61d21ae6f61f7a6b1845c1825/offerings/aro/decisions/adr-001-aro-hcp-alerting-recommendation-for-sre.md#routing-lanes-in-aro-hcp

Discussing with @swiencki to confirm this

Simon has confirmed that's a good lane

machi1990 · 2026-06-25T18:37:58Z

+        hosted_control_plane_version:upgrade_complete:stuck_over_30m:burnrate1h > 14.4
+      for: 5m
+      labels:
+        severity: info


intentionally as sev4 for now and over time it'll be graduated to the corresponding severity as par https://github.com/swiencki/architecture/blob/16fbc2fb1ffa04b61d21ae6f61f7a6b1845c1825/offerings/aro/decisions/adr-001-aro-hcp-alerting-recommendation-for-sre.md#burn-rate-tiers-timing-precision

machi1990 · 2026-06-25T18:38:19Z

+        hosted_control_plane_version:upgrade_complete:stuck_over_30m:burnrate6h > 6
+      for: 30m
+      labels:
+        severity: info


same as https://github.com/Azure/ARO-HCP/pull/5752/changes#r3476800035 intentional as info for now and will be graduated over time

machi1990 · 2026-06-25T18:38:34Z

+        hosted_control_plane_version:upgrade_complete:stuck_over_30m:burnrate3d > 1
+      for: 30m
+      labels:
+        severity: info


same as https://github.com/Azure/ARO-HCP/pull/5752/changes#r3476800035 to make it info and graduate over time

machi1990 · 2026-06-25T18:39:01Z

+        hosted_control_plane_version:selection_to_partial:stuck_over_20m:burnrate6h > 6
+      for: 30m
+      labels:
+        severity: info


same comment as https://github.com/Azure/ARO-HCP/pull/5752/changes#r3476800035

machi1990 · 2026-06-25T18:39:10Z

+        hosted_control_plane_version:selection_to_partial:stuck_over_20m:burnrate3d > 1
+      for: 30m
+      labels:
+        severity: info


same comment as https://github.com/Azure/ARO-HCP/pull/5752/changes#r3476800035

swiencki · 2026-06-25T19:05:55Z

+      annotations:
+        summary: "{{ $labels.cluster }}: Cluster upgrade version selection-to-partial error budget burning rapidly (>14.4x)"
+        description: "More than 14.4x of the 99.95% cluster upgrade version selection-to-partial error budget is being consumed over 5m and 1h windows. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Investigate backend_cluster_version_info on affected clusters."
+        runbook_url: "aka.ms/arohcp-runbook-cluster-upgrade"


Needs the https:// prefix to render as a clickable link in IcM: https://aka.ms/arohcp-runbook-cluster-upgrade.

swiencki · 2026-06-25T19:13:39Z

Before the alert goes live we need a runbook / tsg, and shorthand aka.ms url following the format in https://github.com/openshift-online/architecture/pull/79. I or another ARO Classic lead should be able to create that for you.

cgiradkar · 2026-06-26T10:58:40Z

Before the alert goes live we need a runbook / tsg, and shorthand aka.ms url following the format in openshift-online/architecture#79. I or another ARO Classic lead should be able to create that for you.

About the runbook and TSG, I have created a PR for AZure-Docs-Common. Sent you the link, PTAL

Copilot

Pull request overview

This PR adds user-journey observability for HCP cluster upgrades by introducing Prometheus recording rules, burn-rate alerts, promtool tests, and SRE Grafana dashboards for two SLOs: version selection → partial (20m) and upgrade complete (30m).

Changes:

Add recording rules to derive “first-seen” timestamps, latency, fleet totals, stuck counts/rates, and multi-window burn rates for both SLOs.
Add multi-window burn-rate alerting rules (info severity) for both SLOs and wire them into the RP alerting ruleset.
Add two new Grafana dashboards and regenerate the corresponding Azure Monitor Prometheus rule-group Bicep outputs.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
observability/recording-rules-services.yaml	Registers the new cluster-upgrade recording rule definitions for generation.
observability/grafana-dashboards/sre/user-journey/cluster-upgrade-version-selection-to-partial-slo.json	New dashboard for selection→partial SLI/SLO and stuck/burn-rate visualization.
observability/grafana-dashboards/sre/user-journey/cluster-upgrade-complete-slo.json	New dashboard for upgrade-complete SLI/SLO and stuck/burn-rate visualization.
observability/alerts/userJourneyClusterUpgradeVersionSelectionToPartialRecord-prometheusRule.yaml	Recording rules for first-seen timestamps, latency, stuck detection, fleet totals, and burn rates (selection→partial).
observability/alerts/userJourneyClusterUpgradeVersionSelectionToPartialRecord-prometheusRule_test.yaml	Promtool tests for the selection→partial recording rules.
observability/alerts/userJourneyClusterUpgradeVersionSelectionToPartialMonitor-prometheusRule.yaml	Burn-rate alert rules for the selection→partial SLO.
observability/alerts/userJourneyClusterUpgradeVersionSelectionToPartialMonitor-prometheusRule_test.yaml	Promtool tests for the selection→partial burn-rate alerts.
observability/alerts/userJourneyClusterUpgradeCompleteRecord-prometheusRule.yaml	Recording rules for upgrade-complete pending/latency, stuck detection, totals/rates, and burn rates.
observability/alerts/userJourneyClusterUpgradeCompleteRecord-prometheusRule_test.yaml	Promtool tests for the upgrade-complete recording rules.
observability/alerts/userJourneyClusterUpgradeCompleteMonitor-prometheusRule.yaml	Burn-rate alert rules for the upgrade-complete SLO.
observability/alerts/userJourneyClusterUpgradeCompleteMonitor-prometheusRule_test.yaml	Promtool tests for the upgrade-complete burn-rate alerts.
observability/alerts-rp-services.yaml	Wires the new burn-rate alert rules into the RP services ruleset.
dev-infrastructure/modules/metrics/rules/generatedRPPrometheusAlertingRules.bicep	Generated Azure Monitor rule-group resources for the new RP alert rules.
dev-infrastructure/modules/metrics/rules/generatedRecordingRules.bicep	Generated Azure Monitor rule-group resources for the new recording rules.

+        (
+          time() - hosted_control_plane_version:state_first_seen_timestamp{state="desired"}
+        )
+        unless on(cluster, resource_id, subscription_id, cluster_uuid, version) (
+          backend_cluster_version_info{state="completed"} == 1
+        )


+        description: |
+          More than 14.4x of the 99.95% cluster upgrade version selection-to-partial error budget is being consumed. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Investigate backend_cluster_version_info on affected clusters.
+          Window: 1h/5m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 6x of the 99.95% cluster upgrade version selection-to-partial error budget is being consumed. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Investigate backend_cluster_version_info on affected clusters.
+          Window: 6h/30m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          Cluster upgrade version selection-to-partial has been steadily consuming the 99.95% SLO error budget (1x burn). Cluster upgrades are stuck for over 20 minutes without reaching partial. Investigate backend_cluster_version_info on affected clusters.
+          Window: 3d/6h
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 14.4x of the 99.95% cluster upgrade complete error budget is being consumed. Cluster upgrades are stuck for over 30 minutes without reaching completed. Investigate backend_cluster_version_info on affected clusters.
+          Window: 1h/5m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 6x of the 99.95% cluster upgrade complete error budget is being consumed. Cluster upgrades are stuck for over 30 minutes without reaching completed. Investigate backend_cluster_version_info on affected clusters.
+          Window: 6h/30m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          Cluster upgrade complete has been steadily consuming the 99.95% SLO error budget (1x burn). Cluster upgrades are stuck for over 30 minutes without reaching completed. Investigate backend_cluster_version_info on affected clusters.
+          Window: 3d/6h
+          Management Cluster: {{ $labels.cluster }}


+              "Value": "Pending Minutes",
+              "cluster": "Service Cluster",
+              "cluster_uuid": "Cluster UUID",


+              "Value": "Pending Minutes",
+              "cluster": "Service Cluster",
+              "cluster_uuid": "Cluster UUID",


machi1990 · 2026-06-29T08:57:03Z

/test mega-linter

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

  - recording-rules/access-cluster-slo-recordingRule.yaml
+  - alerts/userJourneyClusterUpgradeRecord-prometheusRule.yaml


+  "title": "Cluster Upgrade SLO",
+  "uid": "cluster-upgrade-slo",
+  "version": 1


Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

 prometheusRules:
  rulesFolders:
  - recording-rules/access-cluster-slo-recordingRule.yaml
+  - alerts/userJourneyClusterUpgradeRecord-prometheusRule.yaml


Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

+        description: |
+          More than 14.4x of the 99.95% cluster upgrade reach-partial error budget is being consumed. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on affected clusters.
+          Window: 1h/5m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 6x of the 99.95% cluster upgrade reach-partial error budget is being consumed. Cluster upgrades are stuck for over 20 minutes without the target version reaching partial. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on affected clusters.
+          Window: 6h/30m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          Cluster upgrade reach-partial has been steadily consuming the 99.95% SLO error budget (1x burn). Cluster upgrades are stuck for over 20 minutes without reaching partial. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on affected clusters.
+          Window: 3d/6h
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 14.4x of the 99.95% cluster upgrade complete error budget is being consumed. Cluster upgrades are stuck for over 30 minutes without reaching completed. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on affected clusters.
+          Window: 1h/5m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          More than 6x of the 99.95% cluster upgrade complete error budget is being consumed. Cluster upgrades are stuck for over 30 minutes without reaching completed. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on affected clusters.
+          Window: 6h/30m
+          Management Cluster: {{ $labels.cluster }}


+        description: |
+          Cluster upgrade complete has been steadily consuming the 99.95% SLO error budget (1x burn). Cluster upgrades are stuck for over 30 minutes without reaching completed. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on affected clusters.
+          Window: 3d/6h
+          Management Cluster: {{ $labels.cluster }}


Track control plane version upgrade activation against a 99.95% / 30m SLO from backend_cluster_version_info. Register recording rules for Azure Monitor and add multi-window burn-rate alerts at Sev 4. Co-authored-by: Cursor <cursoragent@cursor.com> feat: add cluster version selection-to-active SLO Grafana dashboard Fleet-wide SLI view for upgrade activation latency, stuck rate, and error budget consumption. Co-authored-by: Cursor <cursoragent@cursor.com> feat: clarify cluster upgrade naming in version selection-to-active SLO Rename alerts, summaries, and dashboard copy to explicitly describe cluster upgrade version selection-to-active so on-call context is clearer. Co-authored-by: Cursor <cursoragent@cursor.com> feat: add UJ cluster upgrade partial and complete SLO observability Introduce user-journey recording rules, burn-rate alerts, promtool tests, and Grafana dashboards for selection-to-partial (20m) and upgrade-complete SLOs, replacing the earlier selection-to-active prototype with UJClusterUpgrade naming aligned to cluster-upgrade dashboards and slo labels. Co-authored-by: Cursor <cursoragent@cursor.com> refactor: make runbook as TBD refactor: rename shared cluster version state_first_seen_timestamp metric Drop selection_to_partial from the first-seen recording rule name since it tracks desired, partial, and completed states and is shared by both upgrade SLOs. Co-authored-by: Cursor <cursoragent@cursor.com> refactor: align cluster upgrade SLO alerts with RP lane conventions Use long-window-first alert tier suffixes and aka.ms/arohcp-runbook-cluster-upgrade runbook URLs so RP lane monitors match team naming and review requirements. Co-authored-by: Cursor <cursoragent@cursor.com> refactor: rename the recording rules following PR reviews Uses https://prometheus.io/docs/practices/rules/#naming for guidance Co-authored-by: Cursor <cursoragent@cursor.com>

Only consider clusters that have 2 distict version labels to effectively ignore clusters that are under install.

Replace fleet burn-rate alerting with per-upgrade duration alerts at low upgrade volumes, and trim observability to operational essentials. Addresses review feedback that regional burn-rate ratios are equivalent to single-failure detection until upgrade volume supports meaningful denominators: Azure#5752 (comment) - Merge reach-partial and complete monitors into one PrometheusRule - Add userJourneyClusterUpgradeStuckInDesired and StuckInProgress alerts - Drop burn-rate recording rules; keep duration and in-progress metrics - Slim dashboard to in-flight table, stuck counts, and detail tables

machi1990 · 2026-07-08T15:37:19Z

/hold cancel

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

+  # https://github.com/Azure/ARO-HCP/blob/main/backend/pkg/controllers/operationcontrollers/operation_cluster_create.go#L316-L320
+  #
+  # Naming: level:metric:operations per https://prometheus.io/docs/practices/rules/#naming
+  - name: user-journey-cluster-upgrade-recording-rules


+  # completed; missing console component in external auth prevents CVO from marking the
+  # upgrade complete. The complete alert may therefore fire due to missing data plane
+  # configuration rather than control plane failure. The runbook accommodates this.
+  - name: rp-user-journey-cluster-upgrade-monitor-rules


machi1990 · 2026-07-08T15:44:22Z

+        runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade
+        correlationId: "userJourneyClusterUpgradeStuckInDesired/{{ $labels.resource_id }}/{{ $labels.version }}"


The redirection is there and there is WIP PR being worked on

+        runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade
+        correlationId: "userJourneyClusterUpgradeStuckInProgress/{{ $labels.resource_id }}/{{ $labels.version }}"


machi1990 · 2026-07-08T15:43:01Z

+        "content": "### Cluster Upgrade SLOs\n\nPer-upgrade observability from `backend_cluster_version_info` via `hosted_control_plane_upgrade:*` recording rules.\n\n| SLO | Target | Alert |\n|-----|--------|-------|\n| **Reach partial** | `partial` within 20m | `userJourneyClusterUpgradeStuckInDesired` (>20m in `desired`) |\n| **Upgrade complete** | `completed` within 30m | `userJourneyClusterUpgradeStuckInProgress` (>30m in progress) |\n\n**Runbook:** [https://aka.ms/arohcp-runbook-cluster-upgrade](https://aka.ms/arohcp-runbook-cluster-upgrade)",
+        "mode": "markdown"


The redirection is there and the PR is being worked on.

be sure to hold this PR until the runbook merges - you are sending this alert to the production SRE queue. The runbook must resolve and be complete before ICM alerts go to the SRE team.

+  "title": "Cluster Upgrade SLO",
+  "uid": "cluster-upgrade-slo",


Replace fleet burn-rate alerting with per-upgrade duration alerts at low upgrade volumes, and trim observability to operational essentials. Addresses review feedback that regional burn-rate ratios are equivalent to single-failure detection until upgrade volume supports meaningful denominators: Azure#5752 (comment) - Merge reach-partial and complete monitors into one PrometheusRule - Add userJourneyClusterUpgradeStuckInDesired and StuckInProgress alerts - Drop burn-rate recording rules; keep duration and in-progress metrics - Slim dashboard to in-flight table, stuck counts, and detail tables

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

+    - record: hosted_control_plane_upgrade:duration_in_progress:seconds
+      expr: |
+        (
+          time() - hosted_control_plane_upgrade:version_state_first_seen:timestamp{state="desired"}
+        )
+        unless on(cluster, resource_id, subscription_id, cluster_uuid, version) (
+          backend_cluster_version_info{state="completed"} == 1
+        )
+        and on(cluster, resource_id, subscription_id, cluster_uuid, version) (
+          backend_cluster_version_info{state=~"desired|partial"} == 1
+        )


SudoBrendan

We for sure need to fix the leak in the series we're recording.

SudoBrendan · 2026-07-08T17:16:30Z

+      }
+      {
+        record: 'hosted_control_plane_upgrade:version_state_first_seen:timestamp'
+        expression: 'min without (prometheus_replica) (min by (cluster, resource_id, subscription_id, cluster_uuid, version, state) (hosted_control_plane_upgrade:version_state_first_seen:timestamp or (timestamp(backend_cluster_version_info{state=~"desired|partial"} == 1) and on (cluster, resource_id) (hosted_control_plane_upgrade:upgrade_eligible:info == 1)) or timestamp(backend_cluster_version_info{state="completed"} == 1)))'


If I'm reading this correctly - this persists series indefinitely due to the self-reference / recursion. After a single upgrade for a single cluster, we'll have:

{resource_id="foo", version="4.21.0", state="desired"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.0", state="partial"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.0", state="completed"} TIME_FIRST_SEEN

With two upgrades, we'll see double.

{resource_id="foo", version="4.21.0", state="desired"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.0", state="partial"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.0", state="completed"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.1", state="desired"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.1", state="partial"} TIME_FIRST_SEEN
{resource_id="foo", version="4.21.1", state="completed"} TIME_FIRST_SEEN

For 10000 clusters, each upgrading 1/wk (52 times) on z-stream, we'll have 1.5 million different series in a year's time - and nearly all of them will be irrelevant (old upgrades).

Regardless of if the upgrade completed or not - prometheus will continue to track the series and record the data in the interval specified - we'll be up to 1.5 million samples a minute (PT1M) - recording the same metric value for upgrades that completed months/years prior.

We should add a guard to kill old series. What about:

hosted_control_plane_upgrade:version_state_first_seen:timestamp
unless on(cluster, resource_id, subscription_id, cluster_uuid, version) (
backend_cluster_version_info{state="completed"} == 1
)...

In addition - we should probably remove this unused part

or timestamp(backend_cluster_version_info{state="completed"} == 1

This would make it so we only record the most recent upgrade, and only until it's done (no lingering artifacts).

SudoBrendan · 2026-07-08T17:31:48Z

+    - labels: 'hosted_control_plane_upgrade:in_progress:count{cluster="svc-1"}'
+      value: 0
+# ========================================
+# Test 4: Complete — desired transitions to completed within 30m (upgrade)


Adding a test here to verify that the series disappears after upgrade completes would ensure we don't leak now or in the future.

SudoBrendan · 2026-07-08T17:35:10Z

+evaluation_interval: 1m
+tests:
+# ========================================
+# Test 1: Reach partial — desired transitions to partial within 20m (upgrade)


This could also use a test to ensure we get to the state="desired" clause, so we don't fire for "stuck in desired" for things actively progressing.

SudoBrendan · 2026-07-08T17:41:03Z

+  # upgrade complete. The complete alert may therefore fire due to missing data plane
+  # configuration rather than control plane failure. The runbook accommodates this.
+  - name: rp-user-journey-cluster-upgrade-monitor-rules
+    rules:


Alerts should probably define a for, even if it's brief, all others do.

Replace fleet burn-rate alerting with per-upgrade duration alerts at low upgrade volumes, and trim observability to operational essentials. Addresses review feedback that regional burn-rate ratios are equivalent to single-failure detection until upgrade volume supports meaningful denominators: Azure#5752 (comment) - Merge reach-partial and complete monitors into one PrometheusRule - Add userJourneyClusterUpgradeStuckInDesired and StuckInProgress alerts - Drop burn-rate recording rules; keep duration and in-progress metrics - Slim dashboard to in-flight table, stuck counts, and detail tables

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

+          Cluster upgrade target version {{ $labels.version }} has been in desired for over 20 minutes without reaching partial. Partial is when the target version becomes active (recognized as active) on the HCP cluster; the upgrade is in progress but not yet complete. Investigate backend_cluster_version_info on the affected cluster.
+          Service Cluster: {{ $labels.cluster }}
+        runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade
+        correlationId: "userJourneyClusterUpgradeStuckInDesired/{{ $labels.resource_id }}/{{ $labels.version }}"
+    - alert: userJourneyClusterUpgradeStuckInProgress


+          Cluster upgrade target version {{ $labels.version }} has been in progress for over 30 minutes without reaching completed. Completed is when the target version upgrade has finished and is reported as complete on the HCP cluster. Investigate backend_cluster_version_info on the affected cluster.
+          Service Cluster: {{ $labels.cluster }}
+        runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade
+        correlationId: "userJourneyClusterUpgradeStuckInProgress/{{ $labels.resource_id }}/{{ $labels.version }}"


+        "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false },
+        "content": "### Cluster Upgrade SLOs\n\nPer-upgrade observability from `backend_cluster_version_info` via `hosted_control_plane_upgrade:*` recording rules.\n\n| SLO | Target | Alert |\n|-----|--------|-------|\n| **Reach partial** | `partial` within 20m | `userJourneyClusterUpgradeStuckInDesired` (>20m in `desired`) |\n| **Upgrade complete** | `completed` within 30m | `userJourneyClusterUpgradeStuckInProgress` (>30m in progress) |\n\n**Runbook:** [https://aka.ms/arohcp-runbook-cluster-upgrade](https://aka.ms/arohcp-runbook-cluster-upgrade)",
+        "mode": "markdown"
+      },


openshift-ci Bot added the do-not-merge/work-in-progress label Jun 23, 2026

machi1990 force-pushed the control-plane-upgrade-slo-alert branch 2 times, most recently from 0374474 to f13b732 Compare June 24, 2026 08:56

machi1990 changed the title ~~Add cluster version selection-to-active SLO recording rules, alerts & slo dashboard~~ feat: add UJ cluster upgrade partial and complete SLO observability Jun 24, 2026

JameelB reviewed Jun 24, 2026

View reviewed changes

Comment thread dev-infrastructure/modules/metrics/rules/generatedRecordingRules.bicep Outdated

JameelB reviewed Jun 24, 2026

View reviewed changes

swiencki reviewed Jun 24, 2026

View reviewed changes

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 5e3117e to 3896b9f Compare June 25, 2026 11:20

machi1990 commented Jun 25, 2026

View reviewed changes

Comment thread observability/recording-rules-services.yaml Outdated

machi1990 commented Jun 25, 2026

View reviewed changes

swiencki reviewed Jun 25, 2026

View reviewed changes

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 3896b9f to cc93eab Compare June 26, 2026 11:03

swiencki reviewed Jun 26, 2026

View reviewed changes

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from cc93eab to 3a616ab Compare June 29, 2026 07:55

machi1990 marked this pull request as ready for review June 29, 2026 08:36

Copilot AI review requested due to automatic review settings June 29, 2026 08:36

openshift-ci Bot removed the do-not-merge/work-in-progress label Jun 29, 2026

openshift-ci Bot requested review from hbhushan3 and janboll June 29, 2026 08:36

Copilot started reviewing on behalf of machi1990 June 29, 2026 08:36 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 7caa9cc to 408b1b0 Compare July 3, 2026 16:05

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 3, 2026 16:08

Copilot started reviewing on behalf of machi1990 July 3, 2026 16:08 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Comment thread observability/recording-rules-services.yaml Outdated

prometheusRules:

rulesFolders:

- recording-rules/access-cluster-slo-recordingRule.yaml

- alerts/userJourneyClusterUpgradeRecord-prometheusRule.yaml

Copilot AI review requested due to automatic review settings July 6, 2026 09:25

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 567650f to 49b9804 Compare July 6, 2026 09:25

Copilot started reviewing on behalf of machi1990 July 6, 2026 09:25 View session

Copilot AI reviewed Jul 6, 2026

View reviewed changes

machi1990 and others added 3 commits July 8, 2026 17:26

feat: add a recording rule about cluster upgrade eligibility

b115629

Only consider clusters that have 2 distict version labels to effectively ignore clusters that are under install.

refactor: move the recording rules to the dedicated folder

91f3aed

Copilot AI review requested due to automatic review settings July 8, 2026 15:26

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from d178a7e to 685294f Compare July 8, 2026 15:26

Copilot started reviewing on behalf of machi1990 July 8, 2026 15:27 View session

openshift-ci Bot removed the do-not-merge/hold label Jul 8, 2026

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 8, 2026 15:50

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 685294f to 74a9e68 Compare July 8, 2026 15:50

Copilot started reviewing on behalf of machi1990 July 8, 2026 15:51 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

SudoBrendan suggested changes Jul 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 8, 2026 19:41

machi1990 force-pushed the control-plane-upgrade-slo-alert branch from 74a9e68 to e8f39db Compare July 8, 2026 19:41

Copilot started reviewing on behalf of machi1990 July 8, 2026 19:41 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

		- recording-rules/access-cluster-slo-recordingRule.yaml
		- alerts/userJourneyClusterUpgradeRecord-prometheusRule.yaml

		runbook_url: https://aka.ms/arohcp-runbook-cluster-upgrade
		correlationId: "userJourneyClusterUpgradeStuckInDesired/{{ $labels.resource_id }}/{{ $labels.version }}"

		"content": "### Cluster Upgrade SLOs\n\nPer-upgrade observability from `backend_cluster_version_info` via `hosted_control_plane_upgrade:` recording rules.\n\n\| SLO \| Target \| Alert \|\n\|-----\|--------\|-------\|\n\| Reach partial* \| `partial` within 20m \| `userJourneyClusterUpgradeStuckInDesired` (>20m in `desired`) \|\n\| Upgrade complete \| `completed` within 30m \| `userJourneyClusterUpgradeStuckInProgress` (>30m in progress) \|\n\nRunbook: [https://aka.ms/arohcp-runbook-cluster-upgrade](https://aka.ms/arohcp-runbook-cluster-upgrade)",
		"mode": "markdown"

		"title": "Cluster Upgrade SLO",
		"uid": "cluster-upgrade-slo",

Uh oh!

Conversation

machi1990 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Special notes for your reviewer

PR Checklist

Uh oh!

openshift-ci Bot commented Jun 23, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swiencki commented Jun 25, 2026

Uh oh!

cgiradkar commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

machi1990 commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

machi1990 commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

machi1990 commented Jun 23, 2026 •

edited

Loading

SudoBrendan Jul 8, 2026 •

edited

Loading