Add GRPC operation metrics panels to Maestro dashboard by garrickdabbs · Pull Request #6019 · Azure/ARO-HCP

garrickdabbs · 2026-07-10T16:59:29Z

Add GRPC operation metrics panels to Maestro dashboard

Summary

Adds 4 new Grafana panels to the Maestro server dashboard to visualize GRPC server metrics with operation-level granularity. These panels use the new non-breaking _by_operation metrics and have zero impact on existing dashboards.

Changes

Modified Files

observability/grafana-dashboards/maestro/maestro-server.json:

Added 4 new time series panels (IDs 10-13)
Total panels: 9 → 13
No changes to existing panels

New Panels

1. GRPC Request Rate by Operation

Query: sum by (operation, source) (rate(grpc_server_called_total_by_operation{namespace=~"$namespace", type="Publish"}[5m]))
Legend: {{operation}} ({{source}})
Unit: requests/sec
Purpose: Track request rate broken down by operation and source consumer
Position: Row 6, left (gridPos: h=8, w=12, x=0, y=23)

2. GRPC Error Rate by Operation

Query: sum by (operation) (rate(grpc_server_processed_total_by_operation{namespace=~"$namespace", code!="OK"}[5m])) / sum by (operation) (rate(grpc_server_processed_total_by_operation{namespace=~"$namespace"}[5m]))
Legend: {{operation}}
Unit: percent (0-1)
Purpose: Track error rate (non-OK status codes) by operation
Position: Row 6, right (gridPos: h=8, w=12, x=12, y=23)

3. GRPC P95 Latency by Operation

Query: histogram_quantile(0.95, sum by (operation, le) (rate(grpc_server_processed_duration_seconds_by_operation_bucket{namespace=~"$namespace"}[5m])))
Legend: {{operation}}
Unit: seconds
Purpose: Track 95th percentile latency by operation to identify slow operations
Position: Row 7, left (gridPos: h=8, w=12, x=0, y=31)

4. GRPC Message Rate by Operation

Query: sum by (operation, source) (rate(grpc_server_message_received_total_by_operation{namespace=~"$namespace"}[5m]))
Legend: {{operation}} ({{source}})
Unit: messages/sec
Purpose: Track message throughput by operation and source
Position: Row 7, right (gridPos: h=8, w=12, x=12, y=31)

Non-Breaking Metrics Used

These panels use the NEW metrics with _by_operation suffix:

grpc_server_called_total_by_operation
grpc_server_processed_total_by_operation
grpc_server_processed_duration_seconds_by_operation
grpc_server_message_received_total_by_operation

Important: These are NEW metrics that coexist with the old metrics. Existing panels using grpc_server_* (without _by_operation) continue working unchanged.

Dependencies

⚠️ Requires Maestro metrics enhancement: These panels will show "No data" until the Maestro PR is deployed.

Maestro PR: Add operation label to GRPC server metrics openshift-online/maestro#555 (non-breaking)
Deployment: Can deploy independently - no coordination required

Deployment Flexibility

Because this is a non-breaking change:

✅ Can deploy this dashboard PR before Maestro PR (panels show "No data")
✅ Can deploy this dashboard PR after Maestro PR (panels populate immediately)
✅ Can deploy at the same time (ideal)
✅ No downtime or migration window needed

Visual Layout

The dashboard now has 7 rows:

Rows 1-5: Existing panels (unchanged)
Row 6 (NEW): GRPC Request Rate + GRPC Error Rate
Row 7 (NEW): GRPC P95 Latency + GRPC Message Rate

Testing

Verification Steps

Before Maestro deployment: Panels show "No data" (expected, no errors)
After Maestro deployment:
- Import dashboard to Grafana
- Select namespace with Maestro pods
- Verify panels populate with data
- Check operations visible: create, update, delete, subscribe

Expected Behavior

With Maestro metrics deployed:

Panels show data for operations: create, update, delete, subscribe
Legend shows operation types and source consumers
Time series update in real-time

Before Maestro metrics deployed:

Panels show "No data" (expected)
No errors in Grafana logs
Existing panels continue working normally

Benefits

Operation-level visibility: See which operations (Create vs Update vs Delete) are most frequent
Performance analysis: Identify slow operations via P95 latency panel
Error analysis: Track which operations fail most often
Consumer breakdown: See which consumers (CS, etc.) use which operations
Capacity planning: Understand operation distribution for resource sizing
Zero disruption: Existing dashboards unaffected

Example Insights

From GRPC Request Rate panel:

"CS creates 50 resources/min but only deletes 5/min" → Potential leak investigation

From GRPC Error Rate panel:

"Deletes have 5% error rate, creates have 0.1%" → Focus on delete error handling

From GRPC P95 Latency panel:

"Updates take 500ms p95, creates take 50ms p95" → Update operation bottleneck

From GRPC Message Rate panel:

"Subscribe sending 1000 msg/sec" → Throughput visibility

Backwards Compatibility

✅ 100% Backwards Compatible

Existing panels: Continue working unchanged
Existing queries: Not modified
New panels: Use new _by_operation metrics only
No migration: Nothing breaks if Maestro PR isn't deployed yet

Checklist

Screenshots

Screenshots will be added after Maestro metrics are deployed and panels populate with data.

openshift-ci · 2026-07-10T16:59:54Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: garrickdabbs
Once this PR has been reviewed and has the lgtm label, please assign geoberle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

observability/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

This PR updates the Maestro Server Grafana dashboard to add operation-level visibility for Maestro gRPC metrics, introducing new panels intended to break down request volume, error rate, latency, and message throughput by operation (and sometimes source).

Changes:

Added 4 new time series panels (IDs 10–13) to the Maestro Server dashboard for gRPC operation-level observability.
Introduced PromQL queries that group by operation (and source) for request rate, error rate, p95 latency, and message rate.

+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, source) (rate(grpc_server_called_total{namespace=~\"$namespace\", type=\"Publish\"}[5m]))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation) (rate(grpc_server_processed_total{namespace=~\"$namespace\", code!=\"OK\"}[5m])) / sum by (operation) (rate(grpc_server_processed_total{namespace=~\"$namespace\"}[5m]))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.95, sum by (operation, le) (rate(grpc_server_processed_duration_seconds_bucket{namespace=~\"$namespace\"}[5m])))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, source) (rate(grpc_server_message_received_total{namespace=~\"$namespace\"}[5m]))",


+      "title": "GRPC Request Rate by Operation",
+      "type": "timeseries",
+      "description": "Rate of GRPC Publish requests by operation type (create, update, delete) and source consumer"


+      "title": "GRPC Error Rate by Operation",
+      "type": "timeseries",
+      "description": "Error rate (non-OK status codes) by operation type"


+      "title": "GRPC P95 Latency by Operation",
+      "type": "timeseries",
+      "description": "95th percentile request latency by operation type"


+      "title": "GRPC Message Rate by Operation",
+      "type": "timeseries",
+      "description": "Rate of messages received by operation type and source consumer"


Adds 4 new Grafana panels to visualize Maestro GRPC server metrics with operation-level granularity using the new non-breaking metrics. Panels added: 1. GRPC Request Rate by Operation - req/sec by operation and source 2. GRPC Error Rate by Operation - error % by operation 3. GRPC P95 Latency by Operation - 95th percentile latency 4. GRPC Message Rate by Operation - msg/sec by operation and source Uses new non-breaking metrics with '_by_operation' suffix: - grpc_server_called_total_by_operation - grpc_server_processed_total_by_operation - grpc_server_processed_duration_seconds_by_operation - grpc_server_message_received_total_by_operation No impact on existing dashboards - these are NEW panels using NEW metrics. Requires: openshift-online/maestro PR Azure#555 Related: ARO-26754 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, source) (rate(grpc_server_called_total_by_operation{namespace=~\"$namespace\", type=\"Publish\"}[5m]))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation) (rate(grpc_server_processed_total_by_operation{namespace=~\"$namespace\", code!=\"OK\"}[5m])) / sum by (operation) (rate(grpc_server_processed_total_by_operation{namespace=~\"$namespace\"}[5m]))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.95, sum by (operation, le) (rate(grpc_server_processed_duration_seconds_by_operation_bucket{namespace=~\"$namespace\"}[5m])))",


+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, source) (rate(grpc_server_message_received_total_by_operation{namespace=~\"$namespace\"}[5m]))",


openshift-ci · 2026-07-10T17:24:53Z

@garrickdabbs: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/images	`ab4c8dd`	link	true	`/test images`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot AI review requested due to automatic review settings July 10, 2026 16:59

openshift-ci Bot requested review from swiencki and venkateshsredhat July 10, 2026 16:59

Copilot started reviewing on behalf of garrickdabbs July 10, 2026 17:00 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 17:04

garrickdabbs force-pushed the gdabbs/aro-26754-maestro-grpc-dashboard branch from 1635345 to ab4c8dd Compare July 10, 2026 17:05

Copilot started reviewing on behalf of garrickdabbs July 10, 2026 17:05 View session

garrickdabbs mentioned this pull request Jul 10, 2026

Add operation label to GRPC server metrics openshift-online/maestro#555

Draft

10 tasks

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add GRPC operation metrics panels to Maestro dashboard#6019

Add GRPC operation metrics panels to Maestro dashboard#6019
garrickdabbs wants to merge 1 commit into
Azure:mainfrom
garrickdabbs:gdabbs/aro-26754-maestro-grpc-dashboard

garrickdabbs commented Jul 10, 2026 •

edited by openshift-ci Bot

Loading

Uh oh!

openshift-ci Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

openshift-ci Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

garrickdabbs commented Jul 10, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add GRPC operation metrics panels to Maestro dashboard

Summary

Changes

Modified Files

New Panels

1. GRPC Request Rate by Operation

2. GRPC Error Rate by Operation

3. GRPC P95 Latency by Operation

4. GRPC Message Rate by Operation

Non-Breaking Metrics Used

Dependencies

Deployment Flexibility

Visual Layout

Testing

Verification Steps

Expected Behavior

Benefits

Example Insights

Backwards Compatibility

Related

Checklist

Screenshots

Uh oh!

openshift-ci Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

openshift-ci Bot commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

garrickdabbs commented Jul 10, 2026 •

edited by openshift-ci Bot

Loading