Skip to content

Add systemd/launchd supervision and uploader liveness monitoring (fixes #223)#238

Merged
wjllance merged 1 commit into
mainfrom
fix-223-process-supervision
Jul 13, 2026
Merged

Add systemd/launchd supervision and uploader liveness monitoring (fixes #223)#238
wjllance merged 1 commit into
mainfrom
fix-223-process-supervision

Conversation

@wjllance

Copy link
Copy Markdown
Owner

Summary

The maker and its OpenObserve log uploader had no process supervisor. An unexpected death was neither restarted nor alerted; a uploader dying mid-run only surfaced as a stale dashboard; and on SIGKILL/power loss the un-ingested tail of the local log was stranded until a manual catch-up. This PR adds supervision units for both platforms (Linux prod / macOS dev), hardens the wrapper to keep the uploader alive, and adds a boot/crash catch-up — while keeping the design rule that an intentional fail-safe exit is NOT auto-restarted.

Exit-code policy

A fail-safe shutdown is a deliberate, safe stop that needs a human, so it must notify but must not auto-restart; an unexpected death must be restartable. To let supervisors tell them apart, the maker now emits a dedicated exit code.

Exit code Meaning Restart? Notify?
0 Clean stop (Ctrl+C / SIGTERM) No No
75 Intentional maker fail-safe (order-response loss, 3 consecutive cycle errors, position-reconciliation failure, residual-order cleanup failure) No Yes
1 Startup / config / validation error Yes (loop-limited) Yes
101 / signal (137…) Panic, OOM kill, unexpected death Yes Yes

75 is FAIL_SAFE_EXIT_CODE in crates/standx-cli/src/commands/maker/model.rs, returned via a typed FailSafeShutdown error and mapped to the exit code in main. systemd enforces it with RestartPreventExitStatus=75 + OnFailure=; launchd (no per-code exclusion) uses a translator that maps 750 and pairs with KeepAlive.SuccessfulExit=false.

Files added

  • deploy/systemd/standx-maker.service — maker via the observed wrapper (co-runs the uploader); Restart=on-failure, RestartPreventExitStatus=75, OnFailure=standx-notify@, crash-loop limit.
  • deploy/systemd/standx-openobserve-catchup.service — boot oneshot, ordered before the maker.
  • deploy/systemd/standx-notify@.service + standx-notify.sh — OnFailure webhook/journal notifier.
  • deploy/systemd/maker.env.example — EnvironmentFile template.
  • deploy/launchd/com.standx.maker.plist + standx-maker-supervise.sh — exit-code translator + KeepAlive.
  • deploy/launchd/com.standx.openobserve-catchup.plistRunAtLoad catch-up agent.
  • deploy/README.md — install for both platforms + exit-code policy.
  • scripts/openobserve_catchup.sh — idempotent, incremental crash/boot re-upload of the un-ingested tail.

Changed: scripts/run_maker_observed.sh now polls the uploader while the maker runs and relaunches it (with a stderr/webhook notice) if it dies mid-run; Rust exit-code plumbing in main.rs, commands/mod.rs, and the maker model.rs/mod.rs/runtime.rs.

Test plan

  • cargo fmt -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p standx-cli (new unit tests assert fail-safe→75 and generic→1, including the anyhow→Box downcast path).
  • bash -n on every shell script added/edited; plutil -lint on both plists.
  • Smoke-tested the reworked wrapper end-to-end (stub maker, supervision loop, log capture, exit-code propagation) and the catch-up script's skip/no-op paths.
  • Live gate STANDX_ENABLE_LIVE_MAKER untouched; all units default to paper.

Fixes #223

🤖 Generated with Claude Code

systemd + launchd units for the maker and uploader with Restart=on-failure,
an OnFailure notify path, and a distinct fail-safe exit code (75) so an
intentional safe shutdown is never auto-restarted (only notified). Wrapper
now supervises and relaunches the uploader; a boot/crash catch-up re-uploads
any un-ingested log tail.

Rebased onto main after #218-#221 merged. Merged the commands re-export with
#220's panic_webhook_body. Also repairs the same latent main build break that
#231 fixes: #219's equity/margin `notifier.alert(&alert, &symbol)` call did
not update to the 3-arg `await_delivery` form #221 introduced; corrected here
too so this branch compiles standalone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wjllance
wjllance force-pushed the fix-223-process-supervision branch from 0ce4639 to 08dccfd Compare July 13, 2026 08:14
@wjllance
wjllance merged commit 06deb05 into main Jul 13, 2026
7 checks passed
@wjllance
wjllance deleted the fix-223-process-supervision branch July 13, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1][Infra] systemd/launchd 进程守护 + 上传器存活监视

2 participants