Add deadman alert, live-mode webhook enforcement, and panic hook (fixes #220)#232
Merged
Conversation
Silent process death (SIGKILL/OOM/panic/host down) currently notifies nobody: cleanup only runs after the loop breaks normally, OpenObserve has no alert rules, and the in-process webhook is off by default. Close the three reachable gaps from issue #220. 1. Deadman alert: scripts/openobserve_alerts.py provisions an OpenObserve scheduled alert (template + destination + alert, upserted) that fires when the standx_maker stream has no action='cycle_summary' event for ~3 minutes. Mirrors the existing dashboard script's Basic-auth + env var + upsert structure. Documented in docs/15-openobserve.md. 2. Live forced webhook: run_maker now refuses to start under --live when --alert-webhook is unset, or when every alert threshold is 0 (the webhook could never fire). Added alongside the existing live-gate checks; STANDX_ENABLE_LIVE_MAKER lock unchanged. 3. Panic hook: main chains a panic hook that POSTs one last critical notification, reusing the maker webhook payload shape. It runs the blocking POST on a dedicated thread with its own runtime so it is safe to fire mid-unwind, and only when a maker run configured a webhook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #220 (P0 monitoring): when the maker process dies silently (SIGKILL / OOM / panic / host down) nothing notifies anyone — cleanup only runs after the loop breaks normally, OpenObserve has zero alert rules, and the in-process webhook is off by default. This closes the three reachable gaps.
Deadman alert —
scripts/openobserve_alerts.pyprovisions an OpenObserve scheduled alert (template + destination + alert, all upserted) that fires when thestandx_makerstream has had noaction='cycle_summary'event for ~3 minutes (OPENOBSERVE_ALERT_MINUTES, default 3). It POSTs toOPENOBSERVE_ALERT_WEBHOOK. Structure/auth/env vars mirror the existingscripts/openobserve_dashboard.py. This push channel is decoupled from the maker process, so it still fires when the process itself can no longer notify. Documented indocs/15-openobserve.md(new section 6).Live forced webhook —
run_makernow refuses to start under--livewhen--alert-webhookis unset (hard error), or when every alert threshold (--alert-loss,--alert-inventory-pct,--alert-position-change-pct,--alert-uptime) is 0 so the webhook could never fire. Added next to the existing live-gate validation. TheSTANDX_ENABLE_LIVE_MAKERlock is unchanged.Panic hook —
mainchains a panic hook (keeping the default hook that prints the panic/backtrace) that POSTs one last critical notification, reusing the maker webhook payload shape viacommands::panic_webhook_body. It only arms when a maker run configured a webhook, and runs the blocking POST on a dedicated thread with its own current-thread runtime so it is safe to fire while the main tokio runtime is mid-unwind. Best-effort and bounded (5s timeout); never re-panics.Test plan
cargo fmt -- --check— cleancargo clippy --workspace --all-targets -- -D warnings— cleancargo test -p standx-cli— 31 passedpython3 -m py_compile scripts/openobserve_alerts.py— OK--livewithout--alert-webhooknow errors before any order path; panic hook only arms for maker runs with a webhook configured.Follow-up
Not implemented here (per issue item 4): moving OpenObserve off the trading host. Today OpenObserve runs on the same box as the maker, so a host-level failure (OOM/hardware/network) can take down both the maker and the very monitor meant to detect its death. The deadman alert still helps for process-level deaths (SIGKILL/panic/OOM of the maker alone), but a truly independent monitor should run off-host. Tracking as a separate infra change.
Fixes #220
🤖 Generated with Claude Code