Skip to content

feat(ops): a page queues its own fix — incident dispatch into FleetCrown - #443

Merged
github-actions[bot] merged 1 commit into
mainfrom
ops/incident-dispatch
Aug 29, 2026
Merged

feat(ops): a page queues its own fix — incident dispatch into FleetCrown#443
github-actions[bot] merged 1 commit into
mainfrom
ops/incident-dispatch

Conversation

@catomean

Copy link
Copy Markdown
Collaborator

George's complaint (2026-08-29): the Telegram bot became a noise firehose — the morning's four failing appcron units re-paged every 30 minutes and none got fixed autonomously, while the FleetCrown box-runner polled an empty queue on the same machine. Being connected to Loki and FleetCrown should mean these problems solve themselves.

What this adds

/opt/monitoring/incident-dispatch.sh — the missing producer between the alerting stack and FleetCrown's dispatch queue:

  • Every unit failure worth paging also queues a remediation run: POST /api/inject (the same call Loki's fc.sh dispatch makes), journal tail embedded in the prompt, notifyOnClose:true so the run's close summary (root cause → action → what remains) is what lands on the phone.
  • One incident is one dispatch: a dispatch:<unit> stamp under the shared state dir, honoured by both detectors (OnFailure notifier and host-check sweep), cleared on recovery so a re-broken unit gets a fresh agent.
  • Unit→project mapping: <app>-app.service by name; appcron-* from the unit's own ExecStart (app names contain dashes); everything else falls back to fleetcrown.
  • Failure modes never cost the page: missing token file, API refusal, unknown project (retried as fleetcrown) all degrade to journal lines. ALERT_DRY_RUN queues nothing.
  • The page text now carries 🤖 fix agent dispatched (<project>); outcome follows — the clause that turns it from act now into read the outcome when it arrives.

Token SSOT: the existing ck_* agent token from /home/openclaw/.openclaw/calendar-drain.env (one place to rotate).

Verification

  • 18 new cases in test-host-alerts.sh — 78/78 green.
  • Mutation-proven: removing the notifier's dispatch call turns 12 cases red (run against a scratch copy, not the repo).
  • Harness hardening: the fake-token env var is set globally so a suite run ON the box can never fall through to the real token file and enqueue a live run (the docker-prune class).

After merge

Run bash scripts/hetzner/install-host-alerts.sh to deploy to bitbaum (idempotent).

🤖 Generated with Claude Code

https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU

The alerting stack got very good at saying 'X is broken' exactly once — and
then a human still had to do the fixing. 2026-08-29 morning: four appcron
units re-paged every 30 minutes, every one fixable by an agent, none fixed by
one, while the FleetCrown box-runner polled an empty queue on the same box.

New /opt/monitoring/incident-dispatch.sh: every unit failure worth paging also
queues a remediation run (POST /api/inject, the same call Loki's fc.sh makes;
journal tail embedded in the prompt; notifyOnClose so the run's close summary
— root cause, action, what remains — is what reaches the phone). One incident
is one dispatch: a dispatch:<unit> stamp shared by both detectors (OnFailure
notifier + host-check sweep), cleared on recovery. Failure modes never cost
the page: missing token, API refusal, unknown project (retried as fleetcrown)
all degrade to journal lines. ALERT_DRY_RUN queues nothing.

The page itself now says '🤖 fix agent dispatched (<project>)' — the clause
that turns it from 'act now' into 'read the outcome when it arrives'.

18 new test cases in test-host-alerts.sh (78 total); mutation-verified: with
the notifier's dispatch call removed, 12 go red.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RmMktaYn8MuJdEqjp5gWU
@github-actions
github-actions Bot merged commit beb1ee1 into main Aug 29, 2026
3 checks passed
@github-actions
github-actions Bot deleted the ops/incident-dispatch branch August 29, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant