feat(runner/slo): multi-window burn-rate evaluation for SLOs

[RamanKharchee]

Summary

Ports coroot's multi-window multi-burn-rate SLO alerting (model/alert.go + watchers/incidents.go on coroot main) into slo_generator. runner/pkg/enrichers/slo.go was originally a port of coroot's auditor/slo.go, but the multi-window machinery did not come across — only the constant 14.4.

What was wrong

• Single window (1h), single threshold. Fired on any hour that burned >14.4x, including an hour that had already recovered, and never fired at all on a slow burn that still exhausts a 30-day budget.
• No data-coverage guard, so a low-traffic service whose exporter went quiet read as a total outage.
• alert_message computed int(burnRate / 3600) — a burn rate divided by seconds-per-hour, always 0. Every message read "within 0 hours". Coroot formats the window there (br.LongWindow / timeseries.Hour); the port had substituted the rate for the window.

What this does

Each rule pairs a long and a short window and fires only when both exceed the threshold — the long window establishes the burn is significant, the short window confirms it is still happening:

Burn rate	Long	Short	Budget consumed
14.4x	1h	5m	2%
6x	6h	15m	5%

Kept in sync with coroot main — note upstream has since dropped the 3x/24h WARNING tier and tightened tier 2's short window from 30m to 15m.

Coverage guard: a window is judged only when ≥50% of it reported data. Coroot counts defined points in a raw timeseries; the agent only had a single increase() scalar, so the window is now evaluated at a window/12 step and defined points counted across good ∪ bad ∪ valid — a workload with zero errors has no filter_bad series at all and must not be penalised for it.

Backward compatibility

burn_rates and severity are additive. Legacy alert/alert_message are still emitted, and are left untouched when the burn-rate pass yields nothing — so a backend that predates burn_rates keeps working, and a Prometheus hiccup degrades the report rather than blanking the SLO. The agent ships per-cluster on the customer's upgrade cadence, so this had to be version-skew safe in both directions.

Type of change

• Bug fix (non-breaking)
• New feature (non-breaking)

Chart version

• No version bump needed (runner code only — chart image tags are bumped in separate chore(chart) commits, matching fix(runner/alerts): stop leaking internal matcher names into finding descriptions #552/fix(chronosphere): retry transient 503 UNAVAILABLE and bound concurrency #548/fix(runner/triggers): treat Forbidden as 'no rollouts' in service_no_endpoints workload sweep #539)

Test plan

10 new unit tests in slo_burnrate_test.go, including the one that motivates the change:

• TestCalcBurnRates_ShortWindowRecoveredDoesNotFire — long window 20x, short window 0.1x ⇒ does not fire. Single-window evaluation alerts here.
• TestCalcBurnRates_BothWindowsOverThresholdFires — 20x/30x ⇒ CRITICAL
• TestCalcBurnRates_LowCoverageSkipsRule — coverage 0.4 ⇒ rule not judged
• TestCalcBurnRates_ValidSeriesDerivesBad — distribution_cut shape (bad = valid − good, coroot's slowF = total − fast)
• TestFormatSLOStatus_UsesLongWindow — regression guard for the within 0 hours bug
• TestAttachBurnRates_EmptyKeepsLegacyFields — version-skew guard

go build ./...    # pass
go vet ./...      # pass
gofmt -l          # clean
go test ./pkg/enrichers/   # ok

Not yet exercised against a live cluster — the burn-rate math is unit-tested, but the extra PromQL fan-out (4 windows x 2 queries per config, vs 2 before) should be watched on a real Prometheus before this goes to prod.

Related issues

Backend counterpart (consumes burn_rates, adds NO_DATA status): nudgebee-enterprise PR — linked separately.
Refs https://github.com/nudgebee/nudgebee-enterprise/issues/28482

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces multi-window multi-burn-rate SLO evaluation, ported from Coroot, to improve alerting accuracy and prevent false alerts on recovered short-term spikes. It adds slo_burnrate.go and corresponding tests. The review feedback highlights a potential bug where partial query failures could lead to incorrect burn rate calculations, and suggests running the window queries in parallel to reduce network latency bottlenecks.

[gemini-code-assist]

If any of the Prometheus queries fail, runQueries silently swallows the error and omits the key from the results map. Proceeding with incomplete results (e.g., missing filter_bad or filter_good) will lead to incorrect burn rate calculations (such as assuming 0 bad events, which falsely clears active alerts). We should check if len(results) < len(queries) and return an error to safely skip the window.

	a := &AppStatsEnricher{q: s.q}
	results := a.runQueries(ctx, queries, startTime, endTime, step, "", "", "")
	if len(results) < len(queries) {
		return nil, fmt.Errorf("slo_generator: some queries failed or returned no data for window %ds", window)
	}

	out := map[string]windowStats{}

[gemini-code-assist]

Add "sync" to the import block to support thread-safe parallel execution of window queries.

Suggested change

	import (
	"context"
	"fmt"
	"math"
	"time"
	)
	import (
	"context"
	"fmt"
	"math"
	"sync"
	"time"
	)

[gemini-code-assist]

The required windows are currently queried sequentially. Since Prometheus queries are network round-trips and can be slow, querying them sequentially introduces a significant latency bottleneck. We should run statsForWindow in parallel for all required windows using a sync.WaitGroup and protect writes to byWindow with a sync.Mutex.

func (s *SLOEnricher) burnRatesByApp(ctx context.Context, cfg sloConfig, endTime time.Time) map[string][]burnRate {
	byWindow := map[int]map[string]windowStats{}
	var mu sync.Mutex
	var wg sync.WaitGroup
	for _, w := range requiredWindows() {
		wg.Add(1)
		go func(w int) {
			defer wg.Done()
			stats, err := s.statsForWindow(ctx, cfg, w, endTime)
			if err != nil {
				return
			}
			mu.Lock()
			byWindow[w] = stats
			mu.Unlock()
		}(w)
	}
	wg.Wait()