Add systemd/launchd supervision and uploader liveness monitoring (fixes #223)#238
Merged
Conversation
systemd + launchd units for the maker and uploader with Restart=on-failure, an OnFailure notify path, and a distinct fail-safe exit code (75) so an intentional safe shutdown is never auto-restarted (only notified). Wrapper now supervises and relaunches the uploader; a boot/crash catch-up re-uploads any un-ingested log tail. Rebased onto main after #218-#221 merged. Merged the commands re-export with #220's panic_webhook_body. Also repairs the same latent main build break that #231 fixes: #219's equity/margin `notifier.alert(&alert, &symbol)` call did not update to the 3-arg `await_delivery` form #221 introduced; corrected here too so this branch compiles standalone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wjllance
force-pushed
the
fix-223-process-supervision
branch
from
July 13, 2026 08:14
0ce4639 to
08dccfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The maker and its OpenObserve log uploader had no process supervisor. An unexpected death was neither restarted nor alerted; a uploader dying mid-run only surfaced as a stale dashboard; and on SIGKILL/power loss the un-ingested tail of the local log was stranded until a manual catch-up. This PR adds supervision units for both platforms (Linux prod / macOS dev), hardens the wrapper to keep the uploader alive, and adds a boot/crash catch-up — while keeping the design rule that an intentional fail-safe exit is NOT auto-restarted.
Exit-code policy
A fail-safe shutdown is a deliberate, safe stop that needs a human, so it must notify but must not auto-restart; an unexpected death must be restartable. To let supervisors tell them apart, the maker now emits a dedicated exit code.
0751101/ signal (137…)75isFAIL_SAFE_EXIT_CODEincrates/standx-cli/src/commands/maker/model.rs, returned via a typedFailSafeShutdownerror and mapped to the exit code inmain. systemd enforces it withRestartPreventExitStatus=75+OnFailure=; launchd (no per-code exclusion) uses a translator that maps75→0and pairs withKeepAlive.SuccessfulExit=false.Files added
deploy/systemd/standx-maker.service— maker via the observed wrapper (co-runs the uploader);Restart=on-failure,RestartPreventExitStatus=75,OnFailure=standx-notify@, crash-loop limit.deploy/systemd/standx-openobserve-catchup.service— boot oneshot, ordered before the maker.deploy/systemd/standx-notify@.service+standx-notify.sh— OnFailure webhook/journal notifier.deploy/systemd/maker.env.example— EnvironmentFile template.deploy/launchd/com.standx.maker.plist+standx-maker-supervise.sh— exit-code translator + KeepAlive.deploy/launchd/com.standx.openobserve-catchup.plist—RunAtLoadcatch-up agent.deploy/README.md— install for both platforms + exit-code policy.scripts/openobserve_catchup.sh— idempotent, incremental crash/boot re-upload of the un-ingested tail.Changed:
scripts/run_maker_observed.shnow polls the uploader while the maker runs and relaunches it (with a stderr/webhook notice) if it dies mid-run; Rust exit-code plumbing inmain.rs,commands/mod.rs, and the makermodel.rs/mod.rs/runtime.rs.Test plan
cargo fmt -- --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test -p standx-cli(new unit tests assert fail-safe→75 and generic→1, including the anyhow→Box downcast path).bash -non every shell script added/edited;plutil -linton both plists.STANDX_ENABLE_LIVE_MAKERuntouched; all units default to paper.Fixes #223
🤖 Generated with Claude Code