Alert on webhook job wait time instead of Celery queue length

[tomhamiltonstubber]

Summary

CHRONOS-3R ("Queue is too long") has fired ~7k times since June without a real incident. The error threshold in task_send_webhooks was the same number as the dispatcher's backpressure ceiling (dispatcher_max_celery_queue = 100), and the dispatcher deliberately holds the Celery queue at that ceiling whenever the branch queues have a backlog. Any small overshoot (retries, a multi-branch cycle) tripped it: ~85% of events in the last 30 days were qlength 101–110, and the max was ~210.

The regular trigger is TC2's 3-hourly update_agency_paid_invoice_milestone_dates job, which sends ~1,120 webhooks for branch 183 (Hermes) at :34 past every third hour. The round-robin is doing its job: over the last 7 days, other branches' p95 lag from Chronos ingest to send during those bursts was ~9s, while branch 183's own was ~74s.

What we did

• Alert on job wait, not queue length (round-robin path). check_job_wait() finds the oldest head job across the branch queues and logs an error when it is older than dispatcher_max_job_wait_seconds (300s). The dispatcher calls it at most once a minute and before the backpressure check, so stalled workers still alert and a long backlog logs one error a minute rather than one per job.
• Keep the queue-length error, at a sensible threshold. task_send_webhooks now errors only above celery_queue_alert_length (1000), which still covers producers that bypass the dispatcher (Bobbin, use_round_robin=False, retries). The observed max is ~210, so this is silent today.
• Stop dispatch cycles overshooting the ceiling. Each cycle is capped at the headroom left under dispatcher_max_celery_queue. batch_limit is now a job_dispatcher_task parameter like the others.

Out of scope

• Shrinking the TC2 burst itself. Hermes treats the 3-hourly job as the periodic sync for 13 company fields (COMPANY_SYNCABLE_FIELDS in hermes app/tc2/process.py), so sending only when pay dates change would stop the other fields syncing. Draft exploring it: tutorcruncher/TutorCruncher2#17921.

Fixes CHRONOS-3R

🤖 Generated with Claude Code