feat: add maestro server queue alerts and panels

[placecol]

https://redhat.atlassian.net/browse/ARO-26755

What

This PR adds two alerts and alerts and two new panels to the maestro dashboard related to queue length metrics.

1. postgres_notification_queue_usage - A gauge for reporting queue usage of the postegres NOTIFY queue. This metric should be at or very near 0 during normal operation, and a value of > 0.5 (1 being the max) will produce a warning alert.
2. workqueue_depth - A gauge reporting the depth of the in-memory work queue used by maestro to process spec and status events.

PostgresSQL Notification Queue Usage and Work Queue Depth panels were also added to the maestro server dashboard:

Why

Issues have been observed with maestro's processing of spec and status events, and it is hoped that these metrics will aide in diagnosing these issues.

Testing

Unit tests were added for the alerts. Integration tests were also added to maestro for the postgres_notification_queue_usage metric.

Manual testing was performed in a personal dev environment to test the dashboards.

PR Checklist

• PR is scoped to a single task (no mixed concerns)
• Title follows Conventional Commits format
• Summary explains the "Why" behind the change
• Linked to relevant ticket/issue
• Screenshots included (if graph/UI/metrics changes)
• Self-reviewed the diff
• CI/CD checks are passing (ignore Tide)
• Draft PR used for WIP (if applicable)
• Commit history is clean (rebased/squashed)
• Tricky code blocks are commented
• Specific reviewers tagged
• All comment threads resolved before merge

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: placecol
Once this PR has been reviewed and has the lgtm label, please assign janboll for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• dev-infrastructure/OWNERS
• maestro/OWNERS
• observability/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

This PR extends Maestro observability by adding alerts and dashboard panels around queue health signals (PostgreSQL NOTIFY queue usage and Maestro controller workqueue depth) to help diagnose stalled event processing.

Changes:

• Added two Maestro alert rules for PostgreSQL notification queue usage and workqueue depth.
• Added promtool-style unit tests for the two new alerts.
• Added two Grafana panels to the Maestro Server dashboard for the new queue metrics.
• Updated the generated Azure Monitor Prometheus rule group Bicep with the new alerts.

Reviewed changes

Copilot reviewed 4 out of 7 changed files in this pull request and generated 4 comments.

File	Description
observability/grafana-dashboards/maestro/maestro-server.json	Adds two new dashboard panels for Postgres NOTIFY queue usage and workqueue depth.
maestro/alerts/maestro-prometheusRule.yaml	Defines two new Prometheus alerting rules for Maestro queue health.
maestro/alerts/maestro-prometheusRule_test.yaml	Adds unit tests validating the new alert rules fire/do-not-fire behavior.
dev-infrastructure/modules/metrics/rules/generatedPrometheusAlertingRules.bicep	Updates the generated Azure Monitor rule group to include the two new Maestro alerts.

[placecol]

@microsoft-github-policy-service agree [company="Red Hat"]

[placecol]

@microsoft-github-policy-service agree company="Red Hat"

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[placecol]

/retest

[openshift-ci]

@placecol: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/observability-prometheus	ca1fa1c	link	true	/test observability-prometheus
ci/prow/image-updater-images	ca1fa1c	link	true	/test image-updater-images
ci/prow/integration	ca1fa1c	link	true	/test integration
ci/prow/test-unit	ca1fa1c	link	true	/test test-unit
ci/prow/config-change-detection	ca1fa1c	link	true	/test config-change-detection
ci/prow/lint	ca1fa1c	link	true	/test lint
ci/prow/e2e-parallel	ca1fa1c	link	true	/test e2e-parallel
ci/prow/periodic-healthcheck-images	ca1fa1c	link	true	/test periodic-healthcheck-images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.