fix(observability): alert on old prod e2e clusters

[mmazur]

https://redhat.atlassian.net/browse/AROSLSRE-1433

What

Adds ProdE2EHCPClusterOlderThan3Hours, a prod-only version of the existing int/stg stale HCP cluster alert. The new alert uses backend_cluster_created_time_seconds and filters by the prod e2e subscription IDs exposed in the metric subscription_id label.

Why

Prod e2e clusters can be left around and should page separately from the existing int/stg stale-cluster alert. Filtering by subscription keeps the alert scoped to prod e2e workloads instead of all prod customer clusters.

Testing

• Ran go run ./... --config-file ../../observability/alerts-sl-services.yaml from tooling/prometheus-rules
• Verified prod Grafana backend_cluster_created_time_seconds has subscription_id, environment, cluster, and resource_id labels in live services-* datasources

Special notes for your reviewer

The requested prod e2e subscriptions were checked against live prod Grafana. Current live series were present for 403d9de9-132b-4974-94a5-5b78bdfa191e; no current series were returned for the other two subscriptions, but the alert covers all three requested IDs.

PR Checklist

• PR is scoped to a single task (no mixed concerns)
• Title follows Conventional Commits format
• Summary explains the "Why" behind the change
• Linked to relevant ticket/issue
• Screenshots included (if graph/UI/metrics changes)
• Self-reviewed the diff
• CI/CD checks are passing (ignore Tide)
• Draft PR used for WIP (if applicable)
• Commit history is clean (rebased/squashed)
• Tricky code blocks are commented
• Specific reviewers tagged
• All comment threads resolved before merge

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mmazur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• dev-infrastructure/OWNERS [mmazur]
• observability/OWNERS [mmazur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Adds a production-only Prometheus alert to page on stale prod E2E HCP clusters (older than 3 hours) by filtering backend_cluster_created_time_seconds on specific prod E2E subscription_id values, and wires it through the generated Azure Monitor Prometheus rule group output.

Changes:

• Added ProdE2EHCPClusterOlderThan3Hours alert scoped to environment="prod" and a fixed allowlist of E2E subscription IDs.
• Added promtool tests covering: fires for each allowed subscription, doesn’t fire for other prod subscriptions, and doesn’t fire before 3 hours.
• Regenerated the corresponding Bicep rule in generatedPrometheusAlertingRules.bicep.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
observability/alerts/hcp-test-clusters-prometheusRule.yaml	Adds the new prod E2E stale-cluster alert expression + annotations.
observability/alerts/hcp-test-clusters-prometheusRule_test.yaml	Adds promtool coverage for the new prod E2E alert behavior.
dev-infrastructure/modules/metrics/rules/generatedPrometheusAlertingRules.bicep	Includes the newly generated Azure Monitor Prometheus rule for the alert.

[mmazur]

/retest

[mmazur]

/retest

[openshift-ci]

@mmazur: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	3e8261e	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.