Production-grade distributed job scheduler built with Go, Kafka, and Kubernetes. Orchestrates millions of HTTP-based jobs across dynamic infrastructure with bucket-based scheduling, predictive autoscaling, and at-least-once delivery guarantees.
Chronos separates concerns across five independent microservices, each horizontally scalable:
┌─────────────────────────────────────────────────────────────────────────────┐
│ CHRONOS PLATFORM │
│ │
│ ┌──────────────────────┐ ┌──────────────────────────────────────┐ │
│ │ FRONTEND │ │ CHRONOS SERVER (Consolidated) │ │
│ │ Next.js / Mantine │ │ 3 replicas, all serve API traffic │ │
│ │ WebSocket client │ │ │ │
│ └──────────┬───────────┘ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ API Server (Gin) │ │ │
│ │ REST / WebSocket │ │ REST · WebSocket │ │ │
│ │ │ └─────────────────────────────────┘ │ │
│ └─────────────┬────────┤ │ │
│ │ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ Scheduler (Leader-elected) │ │ │
│ │ │ │ Bucket-based dispatch │ │ │
│ │ │ └─────────────────────────────────┘ │ │
│ │ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ Retry Engine (Leader-elected) │ │ │
│ │ │ │ Exponential backoff · DLQ │ │ │
│ │ │ └─────────────────────────────────┘ │ │
│ │ │ ┌─────────────────────────────────┐ │ │
│ │ │ │ Notification Dispatcher │ │ │
│ │ │ │ (in-process) │ │ │
│ │ │ └─────────────────────────────────┘ │ │
│ └────────┤ Leader election via Redis locks │ │
│ └──────────────────────────────────────┘ │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌──────────────────────────┐ │
│ │ JOB EXECUTOR │ │ BULK INGESTOR │ │ BULK EXECUTOR │ │
│ │ Deployment │ │ KEDA ScaledJob │ │ Deployment │ │
│ │ KEDA Scaler + │ │ CSV → Kafka │ │ KEDA Scale-to-Zero │ │
│ │ Predictive │ │ per upload │ │ Dispatch + Aggregate │ │
│ │ Scaler │ │ │ └──────────────────────────┘ │
│ └─────────────────┘ └──────────────────┘ │
│ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ │ │
│ │ KAFKA (KRaft) │ │
│ │ bucket-triggers (12p) → Job Executor → job-events │ │
│ │ bulk-records (24p) → Bulk Executor → bulk-results │ │
│ │ job-events → Flink / Spark │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────┐ ┌──────────────────────────────────┐ │
│ │ ANALYTICS LAYER │ │ PREDICTIVE SCALER │ │
│ │ │ │ K8s Custom Controller │ │
│ │ Flink: Real-time SLA detection │ │ Reads Spark forecasts │ │
│ │ Spark: Batch aggregations, │ │ Patches Job Executor replicas │ │
│ │ load forecasting │ │ │ │
│ └─────────────────────────────────┘ └──────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ DATA LAYER │ │
│ │ │ │
│ │ PostgreSQL + TimescaleDB → Job defs, schedules, metrics, forecasts │ │
│ │ MongoDB 7 → Execution logs & audit trail (TTL) │ │
│ │ Redis 7 → Locks, idempotency, cache, pub/sub │ │
│ │ MinIO / S3 → CSV uploads, result files, exports │ │
│ │ Prometheus → Metrics export for external monitoring │ │
│ │ │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
| Layer | Technology | Purpose |
|---|---|---|
| Compute | Go 1.23 | Primary services (server, executors, scaler) |
| API | Gin, REST, WebSocket | HTTP API + real-time progress streaming |
| Messaging | Apache Kafka (KRaft) | Event-driven pipeline (bucket triggers → job dispatch) |
| Primary Storage | PostgreSQL 16 + TimescaleDB | Durable state, time-series job metrics |
| Logs & Audit | MongoDB 7 | High-write-volume execution logs with TTL |
| Cache & Locks | Redis 7 | Leader election locks, idempotency cache, rate limiting |
| Object Storage | MinIO (S3-compatible) | CSV file uploads for bulk jobs |
| Real-time Analytics | Apache Flink 1.20 | Sub-second SLA violation detection |
| Batch Analytics | Apache Spark 3.5 | Load forecasting → predictive autoscaling |
| Orchestration | Kubernetes, Helm, KEDA | Container orchestration, autoscaling, deployments |
| Frontend | Next.js | Dashboard for job monitoring and management |
Jobs are grouped by execution minute into schedule buckets. The Scheduler publishes one bucket_id per fired window to bucket-triggers, decoupling scheduler frequency (O(minutes)) from job count. Job Executor fan-out workers consume buckets, query jobs from Postgres, and republish to job-dispatch for parallel dispatch. This enables independent horizontal scaling of dispatch workers via KEDA.
The Scheduler, Retry Engine, and Monitor each hold a separate Redis SETNX lock across Chronos Server replicas. In steady state, each replica leads one background loop:
- Scheduler: 10s renewal, 30s TTL
- Retry/Monitor: 30s renewal, 90s TTL
This ensures at-most-once scheduling while keeping the system simple (no distributed consensus library).
Every dispatch sends X-Job-Id, X-Execution-Id, and X-Attempt-Number headers. Redis idempotency cache (SET chronos:idem:{execution_id} 1 NX EX 86400) prevents duplicate processing of the same Kafka message. Clients use X-Execution-Id as their dedup key.
- Flink (Java/Maven) handles millisecond-latency streams: SLA violation detection, per-job timers
- Spark (PySpark/Scala) handles windowed aggregations, p95 baseline computation, and load forecasting
- Predictive Scaler (Go K8s controller) reads Spark forecasts and pre-scales Job Executor before load arrives
- PostgreSQL + TimescaleDB: All durable state; uses
FOR UPDATE SKIP LOCKEDfor lock-free concurrent dequeuing - MongoDB: Execution logs, audit trail (high write volume, schema-flexible, TTL indexes)
- Redis: Locks, idempotency, rate limiting (millisecond-latency hot path)
API keys stored as SHA-256 hash in Postgres (raw key shown once). Outbound job credentials are K8s Secret references resolved at runtime — never persisted in the database.
- Cron & One-Time Jobs: Native cron schedule support with drift prevention
- Bulk Record Dispatch: Stream CSV → Kafka → HTTP dispatch via
bulk-executor - Exponential Backoff Retry: Configurable retry windows, DLQ transitions for terminal failures
- Real-Time Monitoring: WebSocket streaming for bulk job progress
- Horizontal Autoscaling: KEDA reactive scaling + Spark-powered predictive scaling
- Multi-Tenant Isolation: API key authentication, per-workspace data separation
- Prometheus Metrics: Production observability with metrics export
- Go 1.23+
- Docker & Docker Compose
- Kind — local Kubernetes
- Tilt — live development
- kubectl — Kubernetes CLI
- Helm — package management
# Create a local Kind cluster and install infrastructure services
make cluster-up
# Start Tilt (watches source code, auto-rebuilds and redeploys)
make dev
# Tilt UI opens at http://localhost:10350 — view logs and statusThe Tilt setup includes PostgreSQL, MongoDB, Redis, Kafka, Flink, Spark, and all Chronos services.
# Start services without Kubernetes
make compose-up
# Build binaries
make build
# Run tests
make testcmd/ Go binary entrypoints
├── chronos-server/ API Server + Scheduler + Retry Engine (5,000 LOC)
├── job-executor/ HTTP dispatcher, KEDA-scalable
├── bulk-ingestor/ CSV streamer, spawned as K8s Job per upload
├── bulk-executor/ Kafka consumer, scale to zero
└── predictive-scaler/ K8s controller, patches Job Executor replicas
internal/
├── api/http/ REST API (Gin router, handlers, middleware)
├── api/ws/ WebSocket handler for progress streaming
├── scheduler/ Bucket-based job scheduler with leader election
├── cron/ Cron expression parsing and drift prevention
├── retry/ Exponential backoff engine, DLQ transitions
├── notification/ In-process notification dispatcher (Slack, PagerDuty, webhooks)
├── executor/ HTTP + Kafka job dispatch
├── bulk/ Streaming CSV parser, result aggregation
├── store/ Data access layers (Postgres, MongoDB, Redis, Kafka, MinIO)
└── telemetry/ Prometheus metrics, logging
flink/ Apache Flink jobs (Java/Maven)
└── sla-monitor/ Real-time SLA violation detection
spark/ Apache Spark jobs (PySpark/Scala)
└── load-forecaster/ Time-series forecasting for predictive scaling
build/
├── package/ Dockerfiles for all microservices
└── migrations/postgres/ SQL migrations (golang-migrate)
deployments/
├── helm/ Helm charts for all services and infrastructure
├── kind/ Kind cluster configuration
└── docker-compose/ Standalone Docker Compose setup
scripts/ Local dev helpers (cluster setup, test runners)
frontend/ Next.js dashboard (job monitoring, schedule management)
Complete documentation is in the Documentation/ folder:
| Document | Purpose |
|---|---|
| System Overview | Architecture diagram, component roles, data flow |
| Core Design Patterns | Why things work this way: bucket scheduling, leader election, idempotency, databases |
| Data Model | Database schemas: PostgreSQL, MongoDB, Redis, Kafka |
| API Reference | All REST endpoints with examples (job definitions, schedules, executions, bulk jobs) |
| Job Execution Flow | Step-by-step: schedule firing → dispatch → execution → retry |
| Bulk Operations Flow | CSV uploads → row processing → aggregation, PARTIAL_FAILURE recovery |
| Kafka Topics & Events | Event schemas, consumer groups, partitioning, performance characteristics |
| Deployment & Local Dev | Kind + Tilt setup, Helm charts, KEDA scaling, troubleshooting |
| Analytics & Alerts | Flink SLA detection, Spark forecasting, Predictive Scaler |