Scheduled Jobs

[chandan-m]

Phase 6 - One-Time Scheduled Jobs & Execution Pipeline

Overview

This PR implements the end-to-end pipeline for One-Time Scheduled Jobs, fulfilling the requirements for Phase 6 of the Chronos scheduler. It introduces the API for scheduling jobs, the leader-based bucket triggering
mechanism, and the distributed execution workers for both HTTP and Kafka-based delivery.
———

Key Features

• One-Time Scheduling API: POST /v1/schedules endpoint to create one-time job executions with support for idempotency keys, custom payloads, and delivery configuration overrides.
• Leader-Only Scheduler: Implements Redis-based leader election (SETNX with heartbeat) to ensure only one replica at a time triggers scheduled buckets.
• Bucket-Based Dispatch: Jobs are grouped into minute-level schedule_buckets. The Scheduler publishes a single message per bucket to Kafka (bucket-triggers), which is then fanned out by workers.
• Scalable Execution Pool:
  • Fan-out Worker: Consumes bucket triggers and publishes individual jobs to the job-dispatch topic.
  • Dispatch Worker: Consumes from job-dispatch and performs the actual delivery (HTTP or Kafka).
• Result Reporting & Retries: Automatic classification of response codes (e.g., 5xx as retryable, 4xx as terminal), state persistence in Postgres, and log streaming to MongoDB.

———

Implementation Details

• API Handler (internal/api/http/handlers):
  • ScheduleHandler: Manages job creation, resolves delivery configurations from JobDefinitions, and ensures jobs are assigned to the correct minute-bucket.
  • ExecutionHandler: Provides read-access to the execution history and status of jobs.
• Scheduler Logic (internal/scheduler):
  • LeaderElection: Distributed locking via Redis to coordinate high-availability background loops.
  • BucketDispatcher: Monitors the schedule_buckets table for pending windows and triggers them via Kafka.
• Execution Pipeline (internal/executor):
  • FanoutConsumer: High-performance worker that moves jobs from the database to the dispatch queue.
  • DispatchConsumer: The "hot path" for execution, supporting configurable worker pools and timeouts.
  • HTTPDispatcher & KafkaDispatcher: Specific implementations for job delivery types.
• Storage Layer (internal/store):
  • Enhanced JobStore with support for atomic status transitions and FOR UPDATE SKIP LOCKED for concurrent dequeuing (where applicable).
  • Kafka producer/consumer wrappers with at-least-once delivery guarantees.

———

Database Migrations

• 0018-0021: Adds support for job delivery configurations, tags with GIN indexing, and denormalized retry configurations to make the execution path self-sufficient (reducing DB lookups during high load).

———

Testing

• Unit Tests: Coverage for the bucket dispatch loop and leadership logic.
• Integration Tests: Added test/integration/job_execution_test.go covering the full flow from API request to HTTP dispatch and result reporting.
• Smoke Tests: Updated scripts/smoke-test-phase6.sh to verify end-to-end functionality in the Kind cluster.

———

Verification

• Ran make test and make lint.
• Verified the flow using make dev and running the Phase 6 smoke test against a live cluster.