feat(observability): add Kusto-only alert lanes#6017
Conversation
Add service and HCP alert generator lanes that route only to the Event Hub action group used for Kusto ingestion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 31aff0de-9d6d-4870-9732-600ab6570a24
Route the service-workspace copy of the imported Kubernetes alerts through the Kusto-only service lane and remove the old SL svc-kube lane. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 31aff0de-9d6d-4870-9732-600ab6570a24
There was a problem hiding this comment.
Pull request overview
This PR introduces “Kusto-only” alert lanes so alerts can be shipped to the Event Hub/Kusto pipeline without also routing through standard IcM paging lanes, and wires these lanes into the monitoring deployment and alert generation tooling.
Changes:
- Adds Kusto-only alert generator configs for services and HCPs, producing dedicated generated Bicep outputs.
- Updates monitoring deployment to include Kusto-only service + HCP alert rule modules using only the Event Hub action group.
- Updates the prometheus-rules Makefile to generate the new lanes and remove the old
sl-svc-kubelane.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tooling/prometheus-rules/Makefile | Adds run-kusto-* generation targets and includes them in alerts and correlation-map inputs. |
| observability/alerts-kusto-services.yaml | Defines the Kusto-only services lane and redirects output to the new generated Bicep file. |
| observability/alerts-kusto-hcps.yaml | Defines the Kusto-only HCP lane (currently empty rule set) and its output Bicep path. |
| dev-infrastructure/templates/monitoring.bicep | Wires in kustoServiceAlerts and kustoHcpAlerts modules using ehActionGroups only. |
| dev-infrastructure/modules/metrics/kusto-service-rules.bicep | Points to the new generated “Kusto-only service” Prometheus rule groups. |
| dev-infrastructure/modules/metrics/kusto-hcp-rules.bicep | New module to deploy generated “Kusto-only HCP” Prometheus rule groups. |
| dev-infrastructure/modules/metrics/rules/generatedKustoOnlyServicePrometheusAlertingRules.bicep | Generated Prometheus rule groups for Kusto-only service alerts. |
| dev-infrastructure/modules/metrics/rules/generatedKustoOnlyHCPPrometheusAlertingRules.bicep | Generated (currently empty) module; needs linter suppression for the unused severityCeiling param. |
| @description('The minimum IcM severity level (highest priority) that alerts can fire at. Alerts more critical than this ceiling will be degraded to this value. 0 means no ceiling.') | ||
| param severityCeiling int = 0 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mmazur, stevekuznetsov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
@mmazur: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Why
Some alerts should be queryable in Kusto without paging a human. The existing Event Hub/Kusto action group was appended to normal IcM lanes, but there was no generic way to author alerts that route only to Kusto.
What
observability/alerts-kusto-services.yamlobservability/alerts-kusto-hcps.yamlalerts-sl-svc-kube.yamlinto the Kusto-only service lane.Validation
make -C observability alertsaz bicep build --file dev-infrastructure/templates/monitoring.bicep --stdoutaz bicep lint --file dev-infrastructure/templates/monitoring.bicepaz bicep build --file dev-infrastructure/modules/metrics/kusto-service-rules.bicep --stdoutaz bicep lint --file dev-infrastructure/modules/metrics/kusto-service-rules.bicepaz bicep build --file dev-infrastructure/modules/metrics/kusto-hcp-rules.bicep --stdoutaz bicep lint --file dev-infrastructure/modules/metrics/kusto-hcp-rules.bicep