Add operation label to GRPC server metrics

[garrickdabbs]

Add operation-level GRPC server metrics (non-breaking)

Summary

Adds new operation-level GRPC server metrics alongside existing metrics to provide detailed observability into Maestro server operations. This addresses the need for operation-level monitoring identified in ARO-26754.

✅ NON-BREAKING CHANGE: Creates NEW metrics with _by_operation suffix instead of modifying existing metrics.

Changes

Modified Files

cmd/maestro/server/metrics_interceptor.go:

• Added 5 new metrics with _by_operation suffix
• Implemented extractOperation() helper to parse CloudEvent type
• Updated interceptors to populate BOTH old and new metrics
• Added imports for CloudEvent type parsing

New Metrics Added (Non-Breaking)

All new metrics include operation label and use _by_operation suffix:

1. grpc_server_called_total_by_operation - Counter

   • Labels: type, source, operation
   • Help: "Total number of RPCs called on the server, by operation type."

2. grpc_server_processed_total_by_operation - Counter

   • Labels: type, source, operation, code
   • Help: "Total number of RPCs processed on the server by operation type, regardless of success or failure."

3. grpc_server_processed_duration_seconds_by_operation - Histogram

   • Labels: type, source, operation
   • Help: "Histogram of the duration of RPCs processed on the server by operation type."

4. grpc_server_message_received_total_by_operation - Counter

   • Labels: type, source, operation
   • Help: "Total number of messages received on the server from agent and client, by operation type."

5. grpc_server_message_sent_total_by_operation - Counter

   • Labels: type, source, operation
   • Help: "Total number of messages sent by the server to agent and client, by operation type."

Existing Metrics (Unchanged, Backwards Compatible)

These metrics continue to work exactly as before:

1. grpc_server_called_total - Counter with labels: type, source
2. grpc_server_processed_total - Counter with labels: type, source, code
3. grpc_server_processed_duration_seconds - Histogram with labels: type, source
4. grpc_server_message_received_total - Counter with labels: type, source
5. grpc_server_message_sent_total - Counter with labels: type, source

Operation Values

Publish method (extracted from CloudEvent type):

• create - Create resource request
• update - Update resource request
• delete - Delete resource request
• resync - Resync status request
• unknown - Parse error fallback

Subscribe method:

• subscribe - Subscribe to resource status updates

Implementation Details

Dual Metrics Population

Both old and new metrics are populated in the same code path:

// Update existing metrics (backwards compatible)
grpcCalledCountMetric.WithLabelValues(t, source).Inc()

// Update new operation-level metrics
grpcCalledCountByOperationMetric.WithLabelValues(t, source, operation).Inc()

Performance Impact

• Negligible overhead: Both metrics updated in the same code execution
• No additional parsing: Operation extracted once and reused
• No latency increase: Metrics population is already very fast

Testing

Build Verification

• ✅ make binary - Successfully compiled (140MB binary)
• ✅ No lint or format issues
• ✅ All imports resolved correctly

Expected Metrics Output

Existing metrics (continue working):

grpc_server_called_total{type="Publish",source="aro-hcp-clusters-service"} 65
grpc_server_processed_total{type="Publish",source="aro-hcp-clusters-service",code="OK"} 65
grpc_server_processed_duration_seconds_sum{type="Publish",source="aro-hcp-clusters-service"} 3.2

New metrics (added):

grpc_server_called_total_by_operation{type="Publish",source="aro-hcp-clusters-service",operation="create"} 42
grpc_server_called_total_by_operation{type="Publish",source="aro-hcp-clusters-service",operation="update"} 18
grpc_server_called_total_by_operation{type="Publish",source="aro-hcp-clusters-service",operation="delete"} 5

grpc_server_processed_total_by_operation{type="Publish",source="aro-hcp-clusters-service",operation="create",code="OK"} 42
grpc_server_processed_duration_seconds_by_operation_sum{type="Publish",source="aro-hcp-clusters-service",operation="create"} 2.1

Backwards Compatibility

✅ 100% Backwards Compatible

• Existing dashboards: Continue working without any changes
• Existing alerts: Continue working without any changes
• Existing queries: Continue working without any changes
• No migration required: Consumers can adopt new metrics at their own pace

Migration Path (Optional)

Consumers can gradually migrate to the new operation-level metrics:

Phase 1 (after this PR merges):

• Existing metrics continue working
• New metrics become available
• Dashboards can start using new metrics

Phase 2 (gradual migration):

• Update dashboards to use _by_operation metrics
• Create new alerts using operation-level detail
• Test and validate

Phase 3 (future, optional):

• Deprecate old metrics (with advance notice)
• Eventually remove old metrics (6-12 months later)

Benefits

1. Operation-level visibility: Distinguish Create vs Update vs Delete performance
2. Targeted debugging: Identify which operations are slow or failing
3. Capacity planning: Understand operation distribution patterns
4. Consumer analysis: Track which consumers use which operations
5. Better alerting: Create operation-specific SLOs
6. Zero disruption: Existing monitoring continues working

Example Use Cases

Which operation is slowest?

histogram_quantile(0.95, 
  sum by (operation, le) (
    rate(grpc_server_processed_duration_seconds_by_operation_bucket[5m])
  )
)

Which consumer creates the most resources?

sum by (source) (
  rate(grpc_server_called_total_by_operation{operation="create"}[5m])
)

Are deletes failing more than creates?

sum by (operation) (
  rate(grpc_server_processed_total_by_operation{code!="OK"}[5m])
) 
/ 
sum by (operation) (
  rate(grpc_server_processed_total_by_operation[5m])
)

Compare old vs new metrics (validation):

# Should be equal:
sum(rate(grpc_server_called_total[5m]))
  ==
sum(rate(grpc_server_called_total_by_operation[5m]))

Deployment

✅ No coordination required - can deploy independently

• Deploy this PR at any time
• Dashboard updates can deploy before or after
• No downtime or migration window needed

Related

• Jira: ARO-26754 - Add Maestro GRPC metrics instrumentation
• Root Cause: ARO-26043 - Cluster provisioning latency investigation
• Dashboard PR: Add GRPC operation metrics panels to Maestro dashboard Azure/ARO-HCP#6019

Reviewers

Please review:

1. ✅ Non-breaking approach (new metrics alongside old)
2. ✅ Operation extraction logic from CloudEvent type
3. ✅ Dual metrics population in interceptors
4. ✅ Naming convention (_by_operation suffix)
5. ✅ Performance impact (minimal - same code path)

Checklist

• Code follows Maestro repository patterns
• Metrics naming follows Prometheus best practices
• Added 5 new metrics with _by_operation suffix
• Existing metrics unchanged and continue working
• Both old and new metrics populated in interceptors
• Build passes successfully
• No new dependencies added
• Non-breaking change - backwards compatible
• No migration required for existing consumers
• No deployment coordination needed

[coderabbitai]

Warning

Review limit reached

@garrickdabbs, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b4edfeb9-3ccb-4114-812d-361d0555998a

📥 Commits

Reviewing files that changed from the base of the PR and between d711b6a and cea1748.

📒 Files selected for processing (1)

• cmd/maestro/server/metrics_interceptor.go

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}