Skip to content

Suppression handles reliably-fixed incidents quietly instead of dropping them (ADR-034) - #76

Merged
CodexManvik merged 1 commit into
masterfrom
feat/quiet-handling
Sep 22, 2026
Merged

CodexManvik merged 1 commit into
masterfrom
feat/quiet-handling

Conversation

@CodexManvik

Copy link
Copy Markdown
Owner

Implements the "fix it quietly, but log it" decision from #74. It's recorded as ADR-034.

The problem

A suppressed Situation was dropped before RCA and action ever saw it, so the fix never ran. The fault the system had learned to fix was the one it would stop fixing. This was latent until #74 made signatures stable.

What changes

Before Now
Correlation, reliably-fixed signature not emitted (dropped) emitted with handling="quiet" + still on situations.suppressed
Correlation reliability map loaded at boot only refreshed every RELIABILITY_REFRESH_SECONDS (60s); a baseline reset keeps it
Action, quiet + HITL playbook n/a (never arrived) re-checks that playbook's real track record on that signature. If ≥ min samples (3) and ≥ threshold (0.8), the approval is waived; otherwise normal approval
Other gates (disabled / reversible / RBAC / denylist) — unchanged; a failed pre-flight becomes a hard stop (no human to weigh it)

"Real" means no dry runs, and no records from before TrainingRecord.mode existed. It's the same rule graduation uses, shared in common/evidence.py, so compose's dry-run posture never waives an approval.

Logged in three places

  1. Log lines in correlation (when a situation is marked) and in action (when the waiver is granted or refused, with the evidence). Both are searchable in Grafana (Grafana + Loki: every service's logs, the metrics and the audit trail at one URL #75), e.g. {service="action"} |= "quiet situation".
  2. Audit records. quiet-handling records quiet:skip-approval (3/3 real runs worked) or quiet:needs-approval (...), and quiet-approved is added on the execute path.
  3. The outcome itself. RemediationOutcome.handling="quiet" drives a quiet chip on the console card, a "handled quietly · no approval needed" note on the result, and a new quietlyHandled KPI.

Switches (Helm env.*, env INTELLIOPS_*)

  • SUPPRESSION_MODE: quiet (default) or drop (the old behaviour).
  • QUIET_SKIP_APPROVAL: true (default), or false to keep "quiet" as a label only.

Also

The console's reason explainer described #74's new unhealthy:no-rollback and unhealthy:rollback-failed outcomes as "nothing executed". That's wrong, because the fix did run. It now says a human must check.

Tests (852 passed, up from 831)

  • test_quiet_handling.py covers the full action path:
    • A proven playbook runs without an approval request, and it's audited.
    • It falls back to a human on too few runs, dry-run history, another playbook's record, a poor success rate, or with the waiver switched off.
    • A failed rehearsal blocks the run.
    • The other gates still apply, normal situations are untouched, and an AUTO playbook is marked quiet.
  • The engine emits in quiet mode and keeps dropping in drop mode. A reset keeps reliability.
  • The runtime refresh ignores dry runs and never raises.
  • The read model carries handling through and counts quietlyHandled.
  • common/evidence.py has its own tests.
  • Console badge checked in mock mode; no console errors.
  • ruff, npm run build and helm lint are clean.

🤖 Generated with Claude Code

… dropping them (ADR-034)

A suppressed Situation used to be dropped before RCA and action, so the fault
the system had learned to fix was the one it would no longer fix.

- Correlation marks, it does not drop: a signature over the suppression
  threshold (and min samples) is emitted with Situation.handling="quiet" and
  still queued on situations.suppressed, with a log line either way.
  SUPPRESSION_MODE=drop restores the old behaviour. Its reliability map now
  refreshes every RELIABILITY_REFRESH_SECONDS instead of only at boot, and a
  baseline reset keeps it.
- Action re-checks before waiving anything: the approval is skipped only if
  THIS playbook's real track record on THIS signature clears the same bar
  (common/evidence.py - dry runs and pre-`mode` records excluded). Otherwise
  it falls back to a normal approval. QUIET_SKIP_APPROVAL=false keeps quiet
  as a label only.
- Only the HITL wait is waived. The disabled/reversible/RBAC/denylist gates
  still apply, and a failed pre-flight rehearsal becomes a hard stop.
- Never out of sight: log lines in correlation and action (searchable in
  Grafana), a `quiet-handling` audit record carrying the evidence
  ("3/3 real runs worked") plus `quiet-approved` on the execute path,
  RemediationOutcome.handling="quiet", a console badge, and a quietlyHandled KPI.
- Console: the reason explainer described the new unhealthy:no-rollback /
  unhealthy:rollback-failed outcomes as "nothing executed"; it now says the
  fix ran and a human must check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 22, 2026 20:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9e8c3c56-669a-456e-8b46-bd82fa713748


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CodexManvik
CodexManvik merged commit d0d6baf into master Sep 22, 2026
6 checks passed
@CodexManvik
CodexManvik deleted the feat/quiet-handling branch September 22, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants