Skip to content

scheduler: publishing loop wedges silently — overdue pending schedules never attempt (no error, attempt_count=0) #279

Open

Description

@ajianaz

Evidence (2026-09-23, prod v0.9.4)

  • Process uptime 12.5 days (started 2026-09-10 21:51 UTC, the 0.9.4 deploy).
  • Zero successful publishes since process start, while 8 schedules fell due between Sep 12–23.
  • All 8 overdue rows: status=pending, attempt_count=0, error=NULL, never entered processing.
  • /api/health reports status: ok the whole time. No crash, no restart.
  • The only publishes that ever fired on recent versions did so immediately after a restart (e.g. a slot scheduled Aug 31 published Sep 1 23:11 UTC right after a restart). Publishing works; the loop does not keep working.

Meaning

The tick task stops being scheduled — silently. Not a publish-path failure (every publish failure mode writes failed + error; none of that happened), not the #107 timezone bug (normalization is in place), not rate limits or tokens (nothing was attempted).

Suspects

  1. tokio_cron_scheduler job task wedges permanently (panic inside the task, or a deadlock). process_due_schedules logs tick errors and "continues", but a panicked/deadlocked task produces no further ticks and no surfacing anywhere (crates/titen-core/src/scheduler.rs:40-50).
  2. New in 0.9.4: in-process comment fetch work (4d3483b) running on the same runtime; a wedged shared lock would starve the tick.
  3. HTTP client timeouts are present (threads_client.rs:25-28), so plain network hang is unlikely.

Proposed fixes

  1. Heartbeat: update a last_tick_at timestamp inside the tick; expose it on /api/health (scheduler: {last_tick_at, seconds_since_tick}) so external monitors can alert on a stalled loop.
  2. Overdue guard: a schedule pending for more than N hours past scheduled_at (with attempt_count=0 and no processing entry) should be auto-marked failed with an explicit error like "scheduler stall detected" — same spirit as reap_stale_schedules for processing.
  3. Root-cause the wedge: add a catch/wrap around the tick task body so a panic resets the job instead of killing the run loop silently.

Related: #232 (reply_to_id), #233 (auto-approve UI), #257 (retry/backoff).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions