Skip to content

Latest commit

 

History

History
243 lines (196 loc) · 12.6 KB

STACK.md

File metadata and controls

243 lines (196 loc) · 12.6 KB

Running all of hookstack

The whole loop — a signal in, a verdict, an investigation, a card a person could act on, and the ledger that accounts for all of it — plus a downstream you can read and a stub model, in one command. Every step and every expected output below was re-verified from a clean slate — no images, no volumes, no ledger history — on a plain local Docker before this page was last rewritten.

docker compose down -v --remove-orphans   # start from nothing (deletes ledgers)
docker compose up -d --build              # relay :8100 · judge :8200 · probe :8088 · sink · stub
you ──► hookrelay :8100 ──► hookjudge :8200 ──► hookrelay ──► sink
        /hook/inbound            (judges)      /hook/judge-notify
        /hook/alertmanager
             │
             └──► hookprobe :8088 (investigates) ──► /hook/probe-notify ──► sink
                  · every front-door event is copied here; the probe itself
                    decides by level (critical/high) what is worth paying for
                  · plain `up` runs it on the `replay` rehearsal: a recorded
                    investigation played back through the real read-only gate
                    and audit, priced at nothing — so the report, the approve
                    press and the execution record are all visible with no key

The investigator is up from the start, on the rehearsal. To make it real, hand it a model (all three, so a rehearsal's name never sits on a real run):

printf 'HOOKPROBE_RUNTIME=claude\nHOOKPROBE_MODEL=claude-opus-5\nANTHROPIC_API_KEY=sk-ant-...\n' >> .env
docker compose up -d --build
where what
http://127.0.0.1:8100/ the pipe's board (alerts · ledger · deliveries · silences · routing · help)
http://127.0.0.1:8200/ the brain's board (board · verdicts · review · help: what it judged, by which route, what it cost, what failed to come back)
http://127.0.0.1:8088/ui the investigator's console (work · sessions · approvals · knowledge · runtime · help)
docker compose logs -f sink what an operator would actually have received

All three read in Chinese or English and follow the system's light or dark; the gear top right switches either.

The pipe's config is hookrelay/examples/stack.yaml, mounted read-only, so up needs no setup step. Point HOOKRELAY_CONFIG_FILE at your own file to replace it.

A stub model is wired in by default. Without one, every event lands on the rule floor and reuse — which only ever follows an ai verdict — never happens, hiding half the cost policy. The stub is not a model: it returns canned verdicts and reports itself as stub-4o-mini. Put real credentials in .env and it stops being used.

Production does not use this file. hookrelay/deploy/docker-compose.prod.yml runs the pipe alone, bound to loopback, joined to the docker network of whatever platform it feeds.

Drive the whole cost policy

fire() { curl -s -X POST "http://127.0.0.1:8100/hook/$1" \
  -H 'content-type: application/json' -d "$2"; echo; }

# 1 firing — pays
fire inbound '{"title":"Payment gateway 5xx rate 8.1%","message":"gateway-2 5xx at 8.1% over 5 minutes","state":"alerting","env":"prod"}'

# 2 the same condition restated — reuses, costs nothing
fire inbound '{"title":"Payment gateway 5xx rate 8.1%","message":"gateway-2 5xx at 8.4% over 5 minutes","state":"alerting","env":"prod"}'

# 3 a different condition, through the Alertmanager door — pays
fire alertmanager '{"status":"firing","commonLabels":{"alertname":"DiskWillFill","env":"prod"},
  "alerts":[{"status":"firing","labels":{"alertname":"DiskWillFill","env":"prod","service":"k8s"},
  "annotations":{"summary":"k8s node disk usage 93%","description":"node-3 /var has 7% free"}}]}'

# 4 it recovers — inherits what its firing was judged to be, costs nothing
fire alertmanager '{"status":"resolved","commonLabels":{"alertname":"DiskWillFill","env":"prod"},
  "alerts":[{"status":"resolved","labels":{"alertname":"DiskWillFill","env":"prod","service":"k8s"},
  "annotations":{"summary":"k8s node disk usage 93%","description":"fell back to 41%"}}]}'

Every fire answers with its routing trace — the first one reads "outcome":"routed","channels":["to-probe","to-judge"]: copied to the investigator's door and sent to the brain, in one decision.

curl -s http://127.0.0.1:8200/status then shows:

judged 4   paid ratio 50.0%   cost $0.000524   returns {'sent': 4}
routes {'ai': 2, 'reuse': 1, 'recovery': 1}

#1 firing    ai        critical  $0.000262  Payment gateway 5xx rate 8.1%
#2 firing    reuse     critical  $0         Payment gateway 5xx rate 8.1%  <- summary identical to #1
#3 firing    ai        high      $0.000262  k8s node disk usage 93%
#4 RECOVERY  recovery  high      $0         k8s node disk usage 93%        <- inherits #3's high

And the pipe's own ledger closes the books: 24 delivered, 0 queued, 0 dead — per front-door event one copy to to-judge and one to to-probe (4 + 4), per judgement one Feishu card and one DingTalk message (4 + 4), per investigation report the same pair (the gateway, its re-fire's follow-up, the disk: 3 × 2), one approve press forwarded to the investigator's action door and one ruling press forwarded to the judge's feedback door — the cards rendered by lark-bridge in webhook mode from the pipe's card model, the markdown by the shipped DingTalk plugin. One judgement reaches every downstream in its own dialect without the pipe knowing either; the sink logs show the same verdict rendered both ways, and the investigator's reports beside them.

Checking it

bash scripts/stack-smoke.sh          # build, drive every route, assert, tear down
KEEP=1 bash scripts/stack-smoke.sh   # leave it up to poke at

Twenty-eight assertions, run in CI by ci-stack, and they come in two kinds that are worth keeping apart:

  • Eight about the DIALECT (assert_dialect.py), read from the pipe's own ledger and the sink: the node took the handover, signed its way back in, returned something the pipe could read and route, and the loop closed. Any async node passes these. Nothing in them reads the node's ledger.
  • Twenty about hookjudge and the sink (assert_stack.py), read from that service's /status and the sink's log: judged counts, the ai/reuse/recovery split, priced tokens, identity, recovery inheritance, and the cards the downstream received — including the approved procedure's verdict. Properties of one implementation.

That split is what makes hookstack's own acceptance test runnable rather than aspirational:

STACK_BRAIN=mine RETURN_DOOR=my-notify BRAIN_CHANNEL=to-mine \
  bash scripts/stack-smoke.sh        # your node, dialect assertions only

Until the two were separated, "swap hookjudge for a twenty-line judge of your own and this is still green" could not pass for anybody — the smoke asserted summary.cost > 0 and a reuse-route count, which a conforming twenty-line node has no reason to have.

They exist at all because neither service's own gate can see the stack: each stays green while the pipe cannot reach the brain, a compose path points at a directory that moved, or a stand-in starts with nothing wired to it.

The two stand-ins

Neither ships. hookrelay/deploy/docker-compose.prod.yml has no trace of them.

replaces lives with
sink the chat bots' incoming webhooks (the bridge posts its rendered Feishu card here, the DingTalk plugin posts straight in) — and, when HOOKPROBE_EVENT_URL is pointed at its /probe-standin, the investigator's door too hookrelay/examples/sink.py — it stands in for the pipe's downstreams
stub-ai the model provider hookjudge/examples/stub_ai.py — it stands in for the brain's model

lark-bridge is not a stand-in: the compose runs the real sidecar in webhook mode (docs/bridge-protocol.md), with no Lark account, rendering the pipe's card model into the Feishu card the sink then prints — the same image production runs as the app. The pipe loads the shipped example plugins (HOOKRELAY_PLUGINS), which is where the dingtalk type comes from.

Both stand-ins are threaded, which is not a detail: the pipe delivers to channels in parallel, and a single-threaded sink served one connection while the other timed out and was retried — so the downstream received the same alert twice. The ledger still said sent. A duplicate notification caused entirely by the toy on the receiving end, now asserted against.

Three things worth watching

Whether this is working correctly comes down to these, and each has genuinely been broken:

  1. #4's importance must equal #3's. If it does not, the recovery never linked to its firing and the recovery route did not run — on call you would see an alert at high and its recovery at medium.
  2. Step 4's text contains no recovery word (the description reads "fell back to 41%"). Alertmanager signals recovery with status: resolved, and level_map turns that into info — so hookrelay/examples/stack.yaml carries status as a field, or the brain sees no sign of recovery at all.
  3. Two different alerts must have two different identities (visible on /status). If they all share one, the brain is parsing a shape the pipe is not sending — and the near-zero paid ratio that produces looks like excellent cost savings.

Upstream conventions

Two that are easy to get wrong, both encoded in hookrelay/examples/stack.yaml:

Array paths use a dotted index. alerts.0.annotations.summary resolves; alerts[0].annotations.summary returns nothing and does it silently, so the title just comes out empty.

Carry the upstream's state as fields.status. See point 2 above. The brain excludes state fields from identity (alongside timestamps), so carrying it does not split the firing/resolved pair into two conditions.

A source that already judged its own signal takes a terminal route. Not every upstream is an alert source. One that has decided for itself — a watcher forwarding "this needs a person", a system that only emits what matters — gains nothing from the brain, and would be judged in alert vocabulary (severity, recovery, flapping) that does not describe it. Give it a signed door of its own and a route with a higher priority and stop: true, straight to the channel you named: it keeps the pipe's ledger, retries and dead letters, and skips the judging. deploy/shadow.yaml's watch source is the worked example, and scripts/post_watch_signal.py posts to one from off-host without the sending machine needing a copy of the secret.

Such a source can still reach the investigator, and should say what kind of thing it is sending: fields.kind: task asks how the work would be done instead of what broke it. level remains the only thing with teeth — the investigator funds critical/high and declines the rest itself, so a signal worth telling somebody about but not worth investigating is simply low.

Down

docker compose down          # keep the ledgers
docker compose down -v       # delete them too — the next `up` starts from nothing

Known gaps

  • reuse saves money, not attention — decided 2026-08-12: keep it that way for now. Step 2 reused the verdict and paid nothing, but the downstream still got an identical card. This posture turns the pipe's dedup OFF on the grounds that the brain owns noise accounting; the brain now does account for it — summary.attention on /status counts interruptions, repeats and the rulings humans gave them, and hookprobe/examples/patrols/weekly-attention-review.md is a weekly patrol that reads it over seven days and says whether the number is moving. That makes the bill legible; it does not stop the duplicate card. The two ways to close it stay on record for the day repeated cards actually hurt: the brain returns a suppression signal on reuse and the pipe drops the delivery (one place decides what is noise, but a suppressed card can hide a genuine escalation), or the pipe deduplicates the return door (simpler, but two components decide what is noise and the brain's ledger stops describing what was delivered).
  • AWS SNS cannot be templated — closed 2026-08-12 by hookrelay/examples/plugins/aws_sns_source.py: the adapter unwraps the JSON-string Message so templates address the inner alarm ({AlarmName} works), answers the SubscriptionConfirmation handshake with SubscribeURL pinned to SNS hosts, and verifies the certificate signatures when cryptography is installed — one loud warning and transport trust when it is not.