Add adaptive alert noise suppression with scoring and dedup windows

[Sruthi-ng]

What does this PR do?

Implements a comprehensive alert noise suppression system that automatically extends deduplication windows for frequently-firing alerts, preventing alert fatigue without manual intervention.

The system tracks alert frequency across three time windows (1h, 24h, 7d) and computes a noise score (0–100) that drives an exponential dedup window multiplier. High-noise alerts are automatically throttled, and severity trends (worsening/improving/stable) are tracked to provide context.

Key Changes

Core Noise Suppression Logic (alerts/base.py)

• _compute_noise_score(): Composite scoring function weighting recent alerts more heavily (1h×20, 24h×2, 7d×0.5), capped at 100
• _compute_severity_trend(): Compares average severity weights of recent vs. older alerts to detect worsening/improving patterns
• _get_adaptive_dedup_window(): Converts noise score to dedup window using exponential formula: min(max, min × 2^(score/25))
  • score=0 → 60 min (baseline)
  • score=25 → 120 min (2×)
  • score=50 → 240 min (4×)
  • score=100 → 480 min (8×, capped)
• _refresh_noise_record(): Recalculates and persists AlertNoiseRecord after each successful dispatch, counting alerts from audit log
• is_alert_deduped(): Enhanced to use adaptive window from noise record instead of fixed window; respects noise_suppression.enabled config flag

Data Model (backend/models.py)

• AlertLog: Added severity column (fail/warn/info) to track alert severity for trend analysis
• AlertNoiseRecord: New table tracking per-(table, alert_type) noise metrics:
  • Rolling counts: count_1h, count_24h, count_7d
  • Computed metrics: noise_score, severity_trend, is_throttled
  • Unique constraint on (table_name, alert_type) for upsert semantics

API Endpoints (backend/routers/alert_noise.py)

New /alerts/noise router with:

• GET / — List all noise records sorted by score (descending)
• GET /summary — High-level stats: total tracked, throttled count, worsening trends, top-5 noisiest
• GET /{table}/{alert_type} — Fetch noise record for specific alert
• POST /{table}/{alert_type}/reset — Zero out score and throttle flag (for post-incident cleanup)
• GET /{table}/{alert_type}/history — Recent AlertLog rows for pattern analysis

Configuration (config/kit.yml)

New alerts.noise_suppression section with:

• enabled: Toggle for the entire system (default: true)
• min_dedup_window_minutes: Baseline window (default: 60)
• max_dedup_window_minutes: Hard ceiling (default: 480)
• auto_throttle_threshold: 24h count that triggers throttling (default: 10)

Database Migration (alembic/versions/003_add_alert_noise_suppression.py)

• Adds severity column to alert_logs
• Creates alert_noise_records table with indexes on table_name and alert_type

Testing

Comprehensive test suite in tests/test_noise_suppression.py (386 lines) covering:

• Noise scoring: Zero counts, low/high rates, cap enforcement, per-window contributions
• Severity trending: Stable/worsening/improving detection, empty group handling
• Adaptive windows: Score-to-window conversion, custom min/max, exponential scaling
• Noise record refresh: Creation, count accuracy, throttle flag logic, idempotent upserts
• Integration: Dedup with adaptive windows, noise suppression toggle, high-noise window extension

All tests use

https://claude.ai/code/session_018pzrXAhbmudF2BCP3xxqVR