fix(ops): one incident is one message ACROSS detectors, not just within one - #435
Merged
Merged
Conversation
…in one 06:45 `⚙️ FAILED UNIT: orangecat-cat-outcomes.service` from the host-check sweep. 06:46 `🔴 UNIT DOWN: orangecat-cat-outcomes.service` from the OnFailure notifier once its grace expired. One incident, two messages — and each detector was individually correct and correctly deduplicated against itself, which is exactly why this survived yesterday's fix. Keying the cooldown per detector still lets N detectors send N messages about one fact. - The cooldown key is now the SUBJECT (the unit), shared between the sweep and the notifier. Whichever notices first speaks; the other finds the claim taken and goes to the journal. Neither has to know the other exists. - Closure is only sent for a failure that was actually announced. `✅ RECOVERED: unit____` reached George yesterday — closure on a message nobody received, naming a state key instead of a unit. Recoveries now name the unit, and say nothing when nothing was said. - The nightly disk GC no longer pages when it succeeds. "freed 2296MB (81% → 78%)" at 00:02 is the box looking after itself and asking nothing of anyone; the disk check already pages above 85%. It still pages when the outcome is NOT routine — space reclaimed but still above the mark, or nothing left to reclaim. Its test suite gained a delivery capture, because with only a logger stub, moving a message from alert() to logger() is indistinguishable from leaving it alone. test:ops 60 passed in the host-alert suite (was 52) and 22 in disk-gc (was 20). Three mutations — sweep keyed on itself, closure for an unannounced failure, routine GC paging again — each turn their suite red. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AqzRcMP1uJzd7Tav5fNxQz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This morning's messages, after yesterday's fix. Three of the four were still avoidable.
🧹 DISK GC: freed 2296MB (81% → 78%)⚙️ FAILED UNIT: orangecat-cat-outcomes.service🔴 UNIT DOWN: orangecat-cat-outcomes.serviceThe duplicate my last PR did not catch
Both detectors were individually correct, and each was correctly deduplicated against itself — which is exactly why keying per detector let one fact produce two messages. The cooldown key is now the subject (the unit), shared between the sweep and the notifier. Whichever notices first speaks; the other finds the claim taken and journals instead. Neither needs to know the other exists.
Closure only for failures that were announced
✅ RECOVERED: unit____reached George yesterday: closure on a message nobody received, naming a state key instead of a unit. Recoveries now name the unit and are sent only if the failure was actually delivered.The nightly GC report
freed 2296MB (81% → 78%)is the box looking after itself. The disk check already pages above 85%, so a GC that lands back under the mark has by definition removed the reason to say anything — journal. It still pages when the outcome is not routine (still above the mark, or nothing left to reclaim). Its test suite gained a delivery capture: with only aloggerstub, moving a message fromalert()tologger()looks identical to leaving it alone.Verification
test:opsbundle green.0 inert.Note on the one real alert
orangecat-cat-outcomesis a false positive and is fixed separately in the orangecat repo. It claims "the cat_action_log write path looks dead", but the log has 10 rows in the window, the newest from 2026-08-25. Its predicate counts onlystatus=completed+create_*rows, then reports a cause it never tested.🤖 Generated with Claude Code
https://claude.ai/code/session_01AqzRcMP1uJzd7Tav5fNxQz