Repository navigation
fix(intergral/alerting): broadcast alerts to peers under HA partitioning - #67
Conversation
- Enable the existing BroadcastAlerts path when ha_scheduler_partitioning_enabled is set, not only for ha_single_node_evaluation - Mitigates false resolved notifications when a firing rule is reassigned and the previous owner's Alertmanager expires its unrefreshed copy - Measured on a 5-node cluster: 5 false resolved deliveries per topology change before, 1 after; residual case not yet diagnosed - No config surface change
- Widen the push branch filter from "*" to "**" in the Go, UI and docker workflows - "*" does not match "/", so feat/... and fix/... branches got no CI at all after the pull_request triggers were dropped in aabd85c
What this PR doesWhat this PR doesWidens the existing alert-broadcast gate to also trigger under HA scheduler partitioning, and fixes CI branch-name globs that silently disabled builds on slash-named branches.
Risk: Low direct blast radius (5/83,962 symbols, x1.0 amplification, 0 execution roots) — change is a boolean-expression widening on an already-exercised code path, but the fix is explicitly described as a mitigation, not a complete fix, so residual false-resolved notifications can still occur under topology changes. Flagged because it touches a contract (Historian) — callers elsewhere may be affected. ⚡ Blast radius4 files · 12 symbols touched · 7 call sites elsewhere in the codebase
Explore this PR's blast radius → · graph: 💰 Agent economicsno agent sessions have reported to this workspace in the last 30 days, so there is no spend to attribute. Set up agent telemetry → Full economics → · by CodeHub |
Two changes, both small.
CI trigger. The push workflows filtered on
branches: [ "*" ], which does not match/. Since thepull_requesttriggers were dropped in aabd85c, everyfeat/…andfix/…branch has therefore had no CI at all, silently — no run has ever existed for a slash-named branch. Widened to"**". This is why the fix is bundled here: without it this branch could not build.False resolved notifications under partitioning. With
ha_scheduler_partitioning_enabled, each peer runs its own Alertmanager and only receives the alerts its own scheduler evaluates. When a topology change reassigns a firing rule, the previous owner is left holding a copy that nothing refreshes; it expires afternextEndsTimeand that peer dispatches a resolved notification for a rule still firing on its new owner.Upstream already has the mechanism for this —
MultiOrgAlertmanager.BroadcastAlerts, built forha_single_node_evaluation— so this just widens the existing gate to cover partitioning, letting the current owner refresh every peer’s copy.Measured on a 5-node cluster with a receiver that actually delivers, hard-killing 2 peers: 5 false resolved notifications delivered before, 1 after, for alerts the API concurrently reported as firing. Duplicate firing notifications were checked and are not a concern — deliveries stay at one per
repeat_interval, deduplicated by the gossiped notification log.This is a mitigation, not a complete fix. Gossip is best effort and the residual case is not yet diagnosed, so a topology change can still emit an occasional false resolved. Flagging that explicitly because it bears on whether the 12.4.8 RC can be promoted. No config surface change — it activates from the existing partitioning flag.