feat(ops): a page queues its own fix — incident dispatch into FleetCrown - #443
Merged
Conversation
The alerting stack got very good at saying 'X is broken' exactly once — and then a human still had to do the fixing. 2026-08-29 morning: four appcron units re-paged every 30 minutes, every one fixable by an agent, none fixed by one, while the FleetCrown box-runner polled an empty queue on the same box. New /opt/monitoring/incident-dispatch.sh: every unit failure worth paging also queues a remediation run (POST /api/inject, the same call Loki's fc.sh makes; journal tail embedded in the prompt; notifyOnClose so the run's close summary — root cause, action, what remains — is what reaches the phone). One incident is one dispatch: a dispatch:<unit> stamp shared by both detectors (OnFailure notifier + host-check sweep), cleared on recovery. Failure modes never cost the page: missing token, API refusal, unknown project (retried as fleetcrown) all degrade to journal lines. ALERT_DRY_RUN queues nothing. The page itself now says '🤖 fix agent dispatched (<project>)' — the clause that turns it from 'act now' into 'read the outcome when it arrives'. 18 new test cases in test-host-alerts.sh (78 total); mutation-verified: with the notifier's dispatch call removed, 12 go red. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
George's complaint (2026-08-29): the Telegram bot became a noise firehose — the morning's four failing appcron units re-paged every 30 minutes and none got fixed autonomously, while the FleetCrown box-runner polled an empty queue on the same machine. Being connected to Loki and FleetCrown should mean these problems solve themselves.
What this adds
/opt/monitoring/incident-dispatch.sh— the missing producer between the alerting stack and FleetCrown's dispatch queue:POST /api/inject(the same call Loki'sfc.sh dispatchmakes), journal tail embedded in the prompt,notifyOnClose:trueso the run's close summary (root cause → action → what remains) is what lands on the phone.dispatch:<unit>stamp under the shared state dir, honoured by both detectors (OnFailure notifier and host-check sweep), cleared on recovery so a re-broken unit gets a fresh agent.<app>-app.serviceby name;appcron-*from the unit's own ExecStart (app names contain dashes); everything else falls back tofleetcrown.fleetcrown) all degrade to journal lines.ALERT_DRY_RUNqueues nothing.🤖 fix agent dispatched (<project>); outcome follows— the clause that turns it from act now into read the outcome when it arrives.Token SSOT: the existing
ck_*agent token from/home/openclaw/.openclaw/calendar-drain.env(one place to rotate).Verification
test-host-alerts.sh— 78/78 green.After merge
Run
bash scripts/hetzner/install-host-alerts.shto deploy to bitbaum (idempotent).🤖 Generated with Claude Code
https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU