Skip to content

fix(intergral/alerting): broadcast alerts to peers under HA partitioning - #67

Merged
jhawksley-intergral merged 2 commits into
12.4.x-intergralfrom
fix/alerting-partition-false-resolved
Aug 26, 2026
Merged

jhawksley-intergral merged 2 commits into
12.4.x-intergralfrom
fix/alerting-partition-false-resolved

Conversation

@jhawksley-intergral

Copy link
Copy Markdown

Two changes, both small.

CI trigger. The push workflows filtered on branches: [ "*" ], which does not match /. Since the pull_request triggers were dropped in aabd85c, every feat/… and fix/… branch has therefore had no CI at all, silently — no run has ever existed for a slash-named branch. Widened to "**". This is why the fix is bundled here: without it this branch could not build.

False resolved notifications under partitioning. With ha_scheduler_partitioning_enabled, each peer runs its own Alertmanager and only receives the alerts its own scheduler evaluates. When a topology change reassigns a firing rule, the previous owner is left holding a copy that nothing refreshes; it expires after nextEndsTime and that peer dispatches a resolved notification for a rule still firing on its new owner.

Upstream already has the mechanism for this — MultiOrgAlertmanager.BroadcastAlerts, built for ha_single_node_evaluation — so this just widens the existing gate to cover partitioning, letting the current owner refresh every peer’s copy.

Measured on a 5-node cluster with a receiver that actually delivers, hard-killing 2 peers: 5 false resolved notifications delivered before, 1 after, for alerts the API concurrently reported as firing. Duplicate firing notifications were checked and are not a concern — deliveries stay at one per repeat_interval, deduplicated by the gossiped notification log.

This is a mitigation, not a complete fix. Gossip is best effort and the residual case is not yet diagnosed, so a topology change can still emit an occasional false resolved. Flagging that explicitly because it bears on whether the 12.4.8 RC can be promoted. No config surface change — it activates from the existing partitioning flag.

- Enable the existing BroadcastAlerts path when ha_scheduler_partitioning_enabled
  is set, not only for ha_single_node_evaluation
- Mitigates false resolved notifications when a firing rule is reassigned and the
  previous owner's Alertmanager expires its unrefreshed copy
- Measured on a 5-node cluster: 5 false resolved deliveries per topology change
  before, 1 after; residual case not yet diagnosed
- No config surface change
- Widen the push branch filter from "*" to "**" in the Go, UI and docker workflows
- "*" does not match "/", so feat/... and fix/... branches got no CI at all after
  the pull_request triggers were dropped in aabd85c
@codehub-agent

codehub-agent Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

What this PR does

What this PR does

Widens the existing alert-broadcast gate to also trigger under HA scheduler partitioning, and fixes CI branch-name globs that silently disabled builds on slash-named branches.

  • NewAlertsRouter's broadcast flag now activates on HASingleNodeEvaluation || HASchedulerPartitioningEnabled, so MultiOrgAlertmanager.BroadcastAlerts also runs when partitioning is enabled, refreshing firing-alert copies on peers that no longer own the rule.
  • Mitigates false "resolved" notifications from stale, unrefreshed alert copies after topology changes (measured 5→1 false resolves per reassignment on a 5-node cluster); not a complete fix — gossip is best-effort and occasional false resolves can still occur.
  • CI workflow branches filters changed from [ "*" ] to [ "**" ] in three workflow files, since * doesn't match branch names containing / (e.g. feat/…, fix/…), which had left those branches without any push-triggered CI.
  • No config surface change; behavior is gated entirely by the pre-existing ha_scheduler_partitioning_enabled flag.

Risk: Low direct blast radius (5/83,962 symbols, x1.0 amplification, 0 execution roots) — change is a boolean-expression widening on an already-exercised code path, but the fix is explicitly described as a mitigation, not a complete fix, so residual false-resolved notifications can still occur under topology changes.

Flagged because it touches a contract (Historian) — callers elsewhere may be affected.

⚡ Blast radius

4 files · 12 symbols touched · 7 call sites elsewhere in the codebase

symbol kind fan-in file
subscribeToFolderChanges Function 2 pkg/services/ngalert/ngalert.go
shouldRun Method 2 pkg/services/ngalert/ngalert.go
configureHistorianBackend Function 2 pkg/services/ngalert/ngalert.go
init Method 1 pkg/services/ngalert/ngalert.go
ApplyStateHistoryFeatureToggles Function 1 pkg/services/ngalert/ngalert.go
createRemoteAlertmanager Function 1 pkg/services/ngalert/ngalert.go

⚠️ Contracts touched (interfaces or fan-in ≥ 10): Historian — callers elsewhere may be affected.

Explore this PR's blast radius → · graph: workspace c3075765a84f · by CodeHub

💰 Agent economics

no agent sessions have reported to this workspace in the last 30 days, so there is no spend to attribute. Set up agent telemetry →

Full economics → · by CodeHub

@jhawksley-intergral jhawksley-intergral changed the title Alerting: broadcast alerts to peers under HA scheduler partitioning fix(intergral/alerting): broadcast alerts to peers under HA partitioning Aug 26, 2026
@jhawksley-intergral
jhawksley-intergral merged commit 922dcde into 12.4.x-intergral Aug 26, 2026
16 of 17 checks passed
@jhawksley-intergral
jhawksley-intergral deleted the fix/alerting-partition-false-resolved branch August 26, 2026 09:38

This branch was successfully deployed

No deployments
Docker Hub — 3d722525 Deployed Aug 26, 2026 by jhawksley-intergral via Docker #302
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant