Suppression handles reliably-fixed incidents quietly instead of dropping them (ADR-034)

architectural.md

@@ -1749,6 +1749,61 @@ surfaced `interrupted:unknown` rather than a silent double execution.
 
 ---
 
+### ADR-034 — Suppression means "handle quietly", not "drop"
+
+**Context.** The closed loop's promise was: a signature the system has reliably fixed stops
+paging people. The implementation did something stronger. A suppressed Situation was never
+emitted, so it never reached RCA or action, and **the fix never ran**. The fault the system
+had learned to fix was the one fault it would no longer fix. This stayed latent for a long
+time, because suppression rarely fired. Signatures depended on how many samples landed in a
+window, and correlation's reliability map loaded only at boot. Once signatures became stable
+(the set of series, not the multiset of samples), suppression started to fire as coded, and
+the gap became real.
+
+**Decision.** Suppression now means **handle quietly**: remediate without paging a human, and
+record that it happened.
+
+- **Correlation marks, it does not drop.** A signature that clears
+  `reliability_suppress_threshold` over at least `reliability_suppress_min_samples` labelled
+  outcomes is emitted with `Situation.handling="quiet"` (an additive field), and is still
+  queued on `situations.suppressed` for the counter. `suppression_mode=drop` keeps the old
+  behaviour. The reliability map now refreshes every `reliability_refresh_seconds` instead of
+  only at boot. Suppression no longer loses fixes, so it no longer has to be starved.
+- **Quiet is a request; action decides.** Correlation's map is per signature, but the risk is
+  per playbook. So action re-checks **that playbook's real track record on that signature**
+  (`common/evidence.py`: at least the same minimum sample count and threshold, with dry runs
+  and records predating `TrainingRecord.mode` excluded). Only if it clears does the HITL
+  approval get waived (`quiet_skip_approval`, default on). Otherwise the situation goes
+  through the normal approval flow.
+- **The waiver is the only thing that changes.** The disabled, reversible, RBAC and
+  destructive-shape gates all still apply. A failed pre-flight rehearsal becomes a hard stop,
+  as it already was for AUTO, because no human is there to weigh it.
+- **Never out of sight.** There are three records of every quiet decision:
+  - a structured log line in correlation when the situation is marked, and in action when the
+    waiver is granted or refused (both searchable in Grafana);
+  - an audit record `quiet-handling` carrying the evidence (e.g. `3/3 real runs worked`),
+    plus `quiet-approved` on the execute path;
+  - `RemediationOutcome.handling="quiet"`, which shows up as a badge on the console card and
+    in the `quietlyHandled` KPI.
+
+**Consequences.** (+) The loop's promise holds as stated: less paging, the same fixes. (+) The
+waiver is self-correcting: a quiet fix that fails is labelled like any other, lowers the
+track record, and the next occurrence goes back to a human. (+) Only real runs count, the same
+rule graduation uses, so compose's dry-run posture never waives an approval. (−) This is a
+second path to unattended execution beside graduation. Graduation is per playbook across every
+signature; this is per (signature, playbook), and so narrower. Both paths use the same evidence
+rule and the same gates. (−) On a real cluster a fault's first `min_samples` occurrences still
+need a human. That is deliberate.
+
+**Alternatives rejected.** *Keep dropping.* It contradicts the feature's own description.
+*Quiet only as a label* (still asking a human). A "quiet" incident that pages someone is not
+quiet, and without anyone watching it would time out and hold the serial action consumer for
+the whole HITL window. That remains available as `quiet_skip_approval=false`. *Trust
+correlation's per-signature figure.* It isn't specific to the playbook RCA picks, and it
+included dry runs.
+
+---
+
 ## 4. Cross-cutting concerns
 
 **Traceability.** A `correlation_id` is threaded through every `AuditRecord`, so one

common/config.py

@@ -22,8 +22,16 @@ class Settings(BaseSettings):
     reliability_suppress_threshold: float = 0.8
     # Labelled outcomes a signature needs before it can be suppressed at all.
     reliability_suppress_min_samples: int = 3
-    # How often RCA re-reads the training store for its runbook track records.
+    # How often RCA and correlation re-read the training store for track records.
     reliability_refresh_seconds: float = 60.0
+    # What suppression does to a signature the system has reliably fixed:
+    # "quiet" (default) still diagnoses and remediates it, without paging a human
+    # when the playbook's own real track record clears the bar below; "drop" is
+    # the historical behaviour - never emitted, so never fixed.
+    suppression_mode: str = "quiet"  # "quiet" | "drop"
+    # Whether a quiet situation whose playbook qualifies may skip the HITL
+    # approval. False keeps quiet handling to logging/labelling only.
+    quiet_skip_approval: bool = True
     graduation_min_successes: int = 3
     # Whether a dry-run SUCCESS counts toward graduating a playbook to AUTO. Off:
     # a simulation always "succeeds", so it is no evidence a fix is safe unattended.

common/contracts.py

@@ -77,6 +77,11 @@ class Situation(BaseModel):
     signature: str
     peak_score: float | None = None  # correlator max z-score for the window
     baseline: dict | None = None  # per-metric {name: {mean, std}} at emit time
+    # "quiet": correlation has seen this signature fixed reliably and asks for it
+    # to be handled without paging a human. A REQUEST, not a permission - action
+    # re-checks the evidence for the specific playbook before skipping approval,
+    # and every other safety gate still applies. Additive; defaults to "normal".
+    handling: str = "normal"  # "normal" | "quiet"
 
 
 class RootCauseHypothesis(BaseModel):
@@ -216,6 +221,9 @@ class RemediationOutcome(BaseModel):
     steps: list[str] = Field(default_factory=list)
     mode: str = "dry_run"  # "dry_run" | "k8s" | "none" (escalated: no executor ran)
     preflight: PreflightResult | None = None
+    # "quiet" when the fix ran without a human approval on the strength of its
+    # track record (see Situation.handling). Additive; defaults to "normal".
+    handling: str = "normal"
 
 
 class AuditRecord(BaseModel):

common/evidence.py

@@ -0,0 +1,71 @@
+"""What counts as evidence that a fix works - shared by correlation and action.
+
+Two decisions rest on a remediation track record:
+
+- correlation marks a situation `handling="quiet"` when its signature has been
+  fixed reliably, and
+- action lets a quiet situation's playbook run WITHOUT a human approval, but
+  only after re-checking that track record for that exact playbook.
+
+Both must judge evidence the same way, or correlation would keep asking for
+quiet handling that action keeps refusing. Hence one module.
+
+Only real runs count. DryRunRemediator + AlwaysHealthyChecker report success
+unconditionally, and records written before TrainingRecord carried `mode` (None)
+cannot be told apart from dry runs - so neither is evidence that a change to a
+real cluster is safe to make unattended.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+_NOT_REAL = frozenset({None, "dry_run", "none"})
+
+
+def _field(record, name):
+    return record.get(name) if isinstance(record, dict) else getattr(record, name, None)
+
+
+def is_real(record) -> bool:
+    return _field(record, "mode") not in _NOT_REAL
+
+
+def real_records(records) -> list:
+    return [r for r in records if is_real(r)]
+
+
+@dataclass(frozen=True)
+class TrackRecord:
+    worked: int
+    total: int
+
+    @property
+    def ratio(self) -> float:
+        return self.worked / self.total if self.total else 0.0
+
+    def __str__(self) -> str:
+        return f"{self.worked}/{self.total} real runs worked"
+
+
+def track_record(records, signature: str, playbook_id: str | None = None) -> TrackRecord:
+    """Real outcomes for `signature`, optionally narrowed to one playbook."""
+    worked = total = 0
+    for r in records:
+        if not is_real(r) or _field(r, "signature") != signature:
+            continue
+        if playbook_id is not None and _field(r, "playbook_id") != playbook_id:
+            continue
+        total += 1
+        if _field(r, "worked"):
+            worked += 1
+    return TrackRecord(worked, total)
+
+
+def quiet_eligible(
+    records, signature: str, playbook_id: str, threshold: float, min_samples: int
+) -> tuple[bool, TrackRecord]:
+    """May this playbook run on this signature without a human approval?"""
+    record = track_record(records, signature, playbook_id)
+    ok = record.total >= max(1, min_samples) and record.ratio >= threshold
+    return ok, record

deploy/k8s/platform/templates/configmap.yaml

@@ -45,6 +45,10 @@ data:
   INTELLIOPS_CORRELATION_ROBUST_WINDOW: {{ .Values.env.CORRELATION_ROBUST_WINDOW | default "128" | quote }}
   INTELLIOPS_CORRELATION_ROBUST_WARMUP: {{ .Values.env.CORRELATION_ROBUST_WARMUP | default "30" | quote }}
   INTELLIOPS_CORRELATION_ROBUST_KEY_BY: {{ .Values.env.CORRELATION_ROBUST_KEY_BY | default "metric" | quote }}
+  # ADR-034: what suppression does to a reliably-fixed signature, and whether a
+  # qualifying playbook may then run without an approval request.
+  INTELLIOPS_SUPPRESSION_MODE: {{ .Values.env.SUPPRESSION_MODE | default "quiet" | quote }}
+  INTELLIOPS_QUIET_SKIP_APPROVAL: {{ .Values.env.QUIET_SKIP_APPROVAL | default "true" | quote }}
   INTELLIOPS_DETECTION_POLICY: {{ .Values.env.DETECTION_POLICY | quote }}
   INTELLIOPS_RUNBOOK_SELECTOR_MODE: {{ .Values.env.RUNBOOK_SELECTOR_MODE | quote }}
   INTELLIOPS_REMEDIATOR_MODE: {{ .Values.env.REMEDIATOR_MODE | quote }}

deploy/k8s/platform/values.yaml

@@ -79,6 +79,11 @@ env:
   # every service into one per metric. Re-tune warm-up before flipping it: a
   # per-series window fills ~5x slower (one sample per poll, not five).
   CORRELATION_ROBUST_KEY_BY: "metric"
+  # ADR-034. "quiet": a reliably-fixed signature is still remediated, without an
+  # approval request when its playbook's REAL track record qualifies (audited +
+  # logged). "drop": the old behaviour - never emitted, never fixed.
+  SUPPRESSION_MODE: "quiet"
+  QUIET_SKIP_APPROVAL: "true"
   GOVERNANCE_MODE: "in_process"     # "http" in values-live — action + governance are
   GOVERNANCE_URL: "http://governance:8000"  # separate pods, so approvals must go over HTTP
   HITL_POLL_TIMEOUT_SECONDS: "30"   # values-live raises this so a human has time to click Approve

flow.md

@@ -107,7 +107,7 @@ instead of waiting out a poll interval ([ADR-018](architectural.md#adr-018--real
 | `telemetry.raw` | ingestion | correlation | `TelemetryEvent` | a normalized signal |
 | `situations.detected` | correlation | rca, **read** | `Situation` (detected) | an alert storm collapsed into one incident |
 | `situations.diagnosed` | rca | action, **read** | `DiagnosedSituation` (situation + hypotheses) | incident with likely cause + suggested fix |
-| `situations.suppressed` | correlation | **read** | `Situation` (suppressed) | a signature that reliably self-heals — detected, then *not* emitted as an incident (closed-loop suppression, made visible for metrics) |
+| `situations.suppressed` | correlation | **read** | `Situation` (suppressed) | a signature the system has reliably fixed. Under the default `suppression_mode=quiet` it is ALSO emitted on `situations.detected` with `handling="quiet"` and remediated without paging a human when the playbook's real track record qualifies; this topic is the record of it ([ADR-034](architectural.md#adr-034--suppression-means-handle-quietly-not-drop)) |
 | `remediation.outcomes` | action | feedback, **read** | `RemediationOutcome` | did the fix work / roll back |
 | `audit.events` | all services | governance | `AuditRecord` | append-only trail (fire-and-forget) |
 

frontend/src/data/mock.ts

@@ -162,16 +162,20 @@ export const situations: Situation[] = [
       { description: "Latency/queueing under load — capacity contention", confidence: 0.55, suggested_runbook_id: "scale-service", confidence_source: "rule" },
     ],
     suggested_runbook_id: "restart-pod",
-    hitl_mode: "auto",
+    hitl_mode: "hitl",
     reversible: true,
     reliability: 0.86,
     suppressed: false,
+    // A signature fixed reliably before: remediated without an approval request
+    // on restart-pod's real track record (still audited) - quiet handling.
+    handling: "quiet",
     outcome: {
       result: "success",
       health_after: "healthy",
-      mode: "dry_run",
+      mode: "k8s",
       steps: ["restart"],
       preflight: { passed: true, detail: "sandbox: clone healthy in 7s", mode: "k8s" },
+      handling: "quiet",
     },
   },
   {

frontend/src/data/types.ts

@@ -64,8 +64,13 @@ export interface SituationOutcome {
     mode: "off" | "k8s";
     sandbox_namespace?: string | null;
   } | null;
+  /** "quiet": the fix ran without anyone being asked, on the playbook's real track record. */
+  handling?: Handling;
 }
 
+/** "quiet" = a reliably-fixed signature, handled without paging a human (and audited). */
+export type Handling = "normal" | "quiet";
+
 export interface Situation {
   id: string; // "sit-" + signature
   signature: string;
@@ -81,6 +86,7 @@ export interface Situation {
   reversible: boolean;
   reliability: number; // per-signature reliability (0..1)
   suppressed: boolean;
+  handling?: Handling; // "quiet" = correlation asked for this to be handled without paging
   outcome?: SituationOutcome; // present once remediation has produced a result
   peak_score?: number | null;
   baseline?: Record<string, { mean: number; std: number }> | null;
@@ -97,6 +103,7 @@ export interface OutcomeRow {
   reason: OutcomeReason;
   ts: number;
   service: string;
+  handling?: Handling;
 }
 
 export interface AuditRow {
@@ -144,6 +151,7 @@ export interface Metrics {
   approvalsPending: number;
   successRate: number; // 0..1
   needsAttention: number; // escalations awaiting a human — excluded from successRate
+  quietlyHandled?: number; // fixes that ran without anyone being asked (proven track record)
 }
 
 export interface MemberEvent {

frontend/src/views/Incidents.tsx

@@ -40,6 +40,10 @@ function failureNote(reason: string | undefined): string {
   if (!reason) return "no outcome detail recorded";
   if (reason === "unhealthy:rolled-back")
     return "the fix ran, health did not recover, and it was rolled back";
+  if (reason === "unhealthy:no-rollback")
+    return "the fix ran and health did not recover; the playbook has no rollback, so a human must check";
+  if (reason === "unhealthy:rollback-failed")
+    return "the fix ran, health did not recover, and the rollback itself FAILED — a human must check";
   if (reason.startsWith("interrupted:"))
     return "the attempt was interrupted mid-flight — whether the cluster changed is UNKNOWN, so a human must check";
   if (reason === "aborted:timeout")
@@ -52,6 +56,18 @@ function failureNote(reason: string | undefined): string {
   return "gate failed closed — nothing executed";
 }
 
+/** A reliably-fixed signature: remediated without paging a human, still audited. */
+function QuietChip() {
+  return (
+    <span
+      className="rounded-md border border-line px-1.5 py-0.5 font-mono text-2xs text-ink-3"
+      title="This signature has been fixed reliably before. If its playbook's real track record qualifies, it runs without an approval request; either way the decision is in the audit log."
+    >
+      quiet
+    </span>
+  );
+}
+
 const METRIC_DOCS: Record<string, { title: string; formula: string; meaning: string }> = {
   noise: {
     title: "Noise reduction",
@@ -343,6 +359,7 @@ export function Incidents({
                         <div className="flex items-center gap-2">
                           <SevChip sev={s.severity} />
                           <StatusChip status={s.status} />
+                          {s.handling === "quiet" && <QuietChip />}
                         </div>
                         <span className="font-mono text-2xs text-ink-3">{timeAgo(s.first_seen)}</span>
                       </div>
@@ -379,6 +396,7 @@ export function Incidents({
                     <div className="flex items-center gap-2">
                       <SevChip sev={shown.severity} />
                       <StatusChip status={shown.status} />
+                      {shown.handling === "quiet" && <QuietChip />}
                     </div>
                     <h2 className="mt-3 text-2xl font-semibold tracking-tight">{shown.title}</h2>
                     <div className="mt-1.5 flex items-center gap-3 font-mono text-2xs text-ink-3">
@@ -523,6 +541,11 @@ export function Incidents({
                         <div className="text-sm font-medium text-ink">
                           Resolved · <span className="font-mono text-sev-ok">{shown.outcome?.health_after ?? "resolved"}</span>
                           {shown.outcome?.mode === "dry_run" && <span className="ml-2 rounded-md bg-surface-2 px-1.5 py-0.5 font-mono text-2xs text-ink-3">dry-run</span>}
+                          {shown.outcome?.handling === "quiet" && (
+                            <span className="ml-2 rounded-md bg-surface-2 px-1.5 py-0.5 font-mono text-2xs text-ink-3">
+                              handled quietly · no approval needed (proven fix, audited)
+                            </span>
+                          )}
                         </div>
                         {shown.outcome?.steps && shown.outcome.steps.length > 0 && (
                           <div className="mt-1 font-mono text-2xs text-ink-3">steps: {shown.outcome.steps.join(" → ")}</div>

services/action/app.py

@@ -138,6 +138,12 @@ async def lifespan(app: FastAPI):
             stop_event,
             make_guard(settings, app.state.bus),
         ),
+        {
+            "training_store": stores.training_store,
+            "quiet_threshold": settings.reliability_suppress_threshold,
+            "quiet_min_samples": settings.reliability_suppress_min_samples,
+            "quiet_skip_approval": settings.quiet_skip_approval,
+        },
     )
     app.state.consumer_stop = stop_event
     app.state.consumer_thread = thread