Run APScheduler in a dedicated sidecar, not inside gunicorn workers#103
Merged
Conversation
The cockpit-snapshot job (~128s) exceeds gunicorn's --timeout 120, so gunicorn SIGABRTs the worker mid-build. That silently drops every 06/09/18/21 Melbourne snapshot, kills whatever live request shared the worker, and — since one scheduler ran per worker — double-fires every job. docker-compose.prod.yml now runs a separate `scheduler` service (rivaflow.scheduler_main) that owns all cron jobs exclusively; the api service sets RIVAFLOW_RUN_SCHEDULER=0 so it no longer starts one. Behavior is unchanged when the env var is unset (dev/sqlite). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the scheduler disease diagnosed during the Wave-2.4 live acceptance (2026-07-10): APScheduler runs inside BOTH gunicorn UvicornWorkers, so the cockpit snapshot build (~128s, grown past
--timeout 120) gets the worker SIGABRT'd (code 134) mid-build — every scheduled 06/09/18/21 snapshot store has been silently failing since the build crossed 120s, each kill blips live requests on that worker, and every job double-fires (one scheduler per worker).Change
rivaflow/scheduler_main.py— standalone blocking entrypoint (python -m rivaflow.scheduler_main): shared logging config, bounded DB wait,start_scheduler(), SIGTERM/SIGINT → clean shutdown. Explicitly does NOT run migrations (the api's start.sh owns those).schedulercompose service — same image + env as the api, no ports,restart: unless-stopped, 768m cap, healthcheck disabled (no HTTP surface).deploy.sh'scompose up -dauto-creates it on deploy.RIVAFLOW_RUN_SCHEDULERgate (settings.RUN_SCHEDULER): api service sets"0"— web workers no longer schedule. Unset ≡"1", so dev/single-process behavior is byte-identical.core/logging_config.py(app + sidecar emit the same JSON format).One topology move kills both failure modes: jobs no longer inherit the web worker's 120s timeout, and the per-worker double-fire is gone.
Verification
rivaflow-scheduler-1up, jobs registered once, the post-deploy warmup build runs to completion and stores a fresh whoop-summary-v2 snapshot, api logs free of scheduler activity and code-134 across the next tick.ISA:
PAI/MEMORY/WORK/whoop-bitter-lesson-audit/ISA-scheduler-sidecar.md.🤖 Generated with Claude Code