Repository navigation
Suppression handles reliably-fixed incidents quietly instead of dropping them (ADR-034) - #76
Merged
Merged
Conversation
… dropping them (ADR-034)
A suppressed Situation used to be dropped before RCA and action, so the fault
the system had learned to fix was the one it would no longer fix.
- Correlation marks, it does not drop: a signature over the suppression
threshold (and min samples) is emitted with Situation.handling="quiet" and
still queued on situations.suppressed, with a log line either way.
SUPPRESSION_MODE=drop restores the old behaviour. Its reliability map now
refreshes every RELIABILITY_REFRESH_SECONDS instead of only at boot, and a
baseline reset keeps it.
- Action re-checks before waiving anything: the approval is skipped only if
THIS playbook's real track record on THIS signature clears the same bar
(common/evidence.py - dry runs and pre-`mode` records excluded). Otherwise
it falls back to a normal approval. QUIET_SKIP_APPROVAL=false keeps quiet
as a label only.
- Only the HITL wait is waived. The disabled/reversible/RBAC/denylist gates
still apply, and a failed pre-flight rehearsal becomes a hard stop.
- Never out of sight: log lines in correlation and action (searchable in
Grafana), a `quiet-handling` audit record carrying the evidence
("3/3 real runs worked") plus `quiet-approved` on the execute path,
RemediationOutcome.handling="quiet", a console badge, and a quietlyHandled KPI.
- Console: the reason explainer described the new unhealthy:no-rollback /
unhealthy:rollback-failed outcomes as "nothing executed"; it now says the
fix ran and a human must check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the "fix it quietly, but log it" decision from #74. It's recorded as ADR-034.
The problem
A suppressed Situation was dropped before RCA and action ever saw it, so the fix never ran. The fault the system had learned to fix was the one it would stop fixing. This was latent until #74 made signatures stable.
What changes
handling="quiet"+ still onsituations.suppressedRELIABILITY_REFRESH_SECONDS(60s); a baseline reset keeps it"Real" means no dry runs, and no records from before
TrainingRecord.modeexisted. It's the same rule graduation uses, shared incommon/evidence.py, so compose's dry-run posture never waives an approval.Logged in three places
{service="action"} |= "quiet situation".quiet-handlingrecordsquiet:skip-approval (3/3 real runs worked)orquiet:needs-approval (...), andquiet-approvedis added on the execute path.RemediationOutcome.handling="quiet"drives aquietchip on the console card, a "handled quietly · no approval needed" note on the result, and a newquietlyHandledKPI.Switches (Helm
env.*, envINTELLIOPS_*)SUPPRESSION_MODE:quiet(default) ordrop(the old behaviour).QUIET_SKIP_APPROVAL:true(default), orfalseto keep "quiet" as a label only.Also
The console's reason explainer described #74's new
unhealthy:no-rollbackandunhealthy:rollback-failedoutcomes as "nothing executed". That's wrong, because the fix did run. It now says a human must check.Tests (852 passed, up from 831)
test_quiet_handling.pycovers the full action path:handlingthrough and countsquietlyHandled.common/evidence.pyhas its own tests.ruff,npm run buildandhelm lintare clean.🤖 Generated with Claude Code