refactor(ops): one voice on the phone — grace, digests, backoff, subjects - #444
Merged
Conversation
…ects
George, 2026-08-29, pasting his Telegram: 'refactor whatever is sending this
abomination.' The morning read as: identical FAILED triples at 08:01, 08:31,
09:01, 09:32, 10:02, 10:33; a six-pack of FAILED+RECOVERED for a 5-minute
deploy blip; closures naming state keys ('RECOVERED:
rtc_revampit_configured_host_does_not_resolve__typing_static_nvi',
'agentwork_reparaturbonus-zh'); UNIT DOWN bodies that repeat the unit name
four times in systemd boilerplate while burying the app's actual error; and
two URL monitors speaking two dialects.
One rule set, all in lib-alert.sh and its callers:
- Reminders BACK OFF: the claim stamp holds 'ts count'; each claim quadruples
the next wait (30m, 2h, 8h, capped at a day). alert_clear resets the
ladder. claim_once is the decision without the delivery, so callers can
digest.
- The sweep gets ONE-TICK GRACE: first sighting watches, never pages — the
OnFailure notifier owns the instant page and already declines deploy blips.
A blip inside the grace produces zero messages, failure or closure.
- DIGESTS: N failures on one tick are one 'STILL FAILING: a; b' message; same
for recoveries.
- Recovery names the SUBJECT, never the key: alert_transition remembers the
human subject at bad-time (explicit arg, else text-to-first-colon, else
key) — fixes every alert_transition caller fleet-wide at one seam.
- The notifier page shows the APP's error: systemd liturgy filtered from the
journal tail; '🔴 DOWN: appcron-kivvi-dunning — curl: (22) … HTTP 500'.
- fleet-uptime.yml speaks the same dialect ('🔴 DOWN: app (…)',
'✅ RECOVERED: app', no ISO timestamp — Telegram stamps messages).
92 tests green; mutation-verified per area: no-grace → 8 red, key-recovery →
3 red, flat-backoff → 1 red.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
George's ask, verbatim: refactor whatever is sending this abomination (his Telegram from the last 14 hours pasted as evidence).
What that feed actually contained, and what changes:
⚙️ STILL FAILING: a; bmessage✅ RECOVERED: rtc_revampit_configured_host_does_not_resolve__typing_static_nvi,agentwork_reparaturbonus-zhalert_transitionprinted its own state key on recoveryUNIT DOWN: appcron-kivvi-dunning.service — appcron-kivvi-dunning.service: Main process exited … Failed to start appcron-kivvi-dunning …🔴 DOWN: appcron-kivvi-dunning — curl: (22) … HTTP 500 → 🤖 fix agent dispatched (kivvi)🔴 kivvi is DOWN (… ) as of 2026-08-28T21:30:35.727Zvs🔴 DOWN: botsmann (…) → HTTP 502Verification
test-host-alerts.sh).After merge
bash scripts/hetzner/install-host-alerts.sh(idempotent) deploys to bitbaum.🤖 Generated with Claude Code
https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU