scaledjob-operator

scaledjob-operator is a Kubernetes operator written in Go that watches a custom resource called ScaledJob and automatically creates Kubernetes Jobs in response to Redis queue depth.

When a queue fills up, the operator creates more Jobs to drain it. When the queue empties, it stops creating Jobs and lets the running ones finish. The operator is the only thing that touches the Jobs, you never scale manually.

The goal is to demonstrate a complete, production-shaped operator: a well-defined CRD, a correct reconcile loop, external state reading, and observable status.

This project was set up using Kubebuilder.

How It Scales

In simple terms: you create a ScaledJob, point it at a Redis list, and provide a Kubernetes Job template. The controller reads the queue depth, calculates how many Jobs should be active, creates owned Jobs, and updates status, metrics, and Kubernetes Events.

threshold: 10
minReplicas: 0
maxReplicas: 5

With this config:

0 messages     -> 0 Jobs
1-10 messages  -> 1 Job
11-20 messages -> 2 Jobs
21-30 messages -> 3 Jobs
41+ messages   -> 5 Jobs max

Completed and failed Jobs do not count as active. The controller does not delete running Jobs when the queue shrinks; they finish normally.

Demo

Local Demo

Prerequisites:

• Docker or another container runtime
• Kind
• kubectl
• redis-cli

Create a cluster and install the CRD:

kind create cluster --name scaledjob-operator-dev
kubectl config use-context kind-scaledjob-operator-dev
make install

Run Redis in the cluster:

kubectl create deployment redis --image=redis:7
kubectl expose deployment redis --port=6379
kubectl rollout status deployment/redis

Forward Redis to your machine. Keep this running:

kubectl port-forward svc/redis 6380:6379

Run the operator locally in another terminal:

make run

Apply the sample:

kubectl apply -k config/samples/

Push 30 messages. With threshold: 10 and maxReplicas: 5, the operator targets ceil(30/10) = 3 Jobs:

redis-cli -p 6380 LPUSH demo-queue \
  m1 m2 m3 m4 m5 m6 m7 m8 m9 m10 \
  m11 m12 m13 m14 m15 m16 m17 m18 m19 m20 \
  m21 m22 m23 m24 m25 m26 m27 m28 m29 m30

Watch the operator respond (within ~5 seconds):

kubectl get scaledjobs

kubectl get jobs

kubectl describe scaledjob scaledjob-sample

Note: each worker pops one message then sleeps ten seconds before completing. The queue does not drain automatically — after all three Jobs finish, the controller reconciles again and creates three more until the queue is empty. To watch the full drain, leave the operator running and monitor with:

watch -n5 'redis-cli -p 6380 LLEN demo-queue && kubectl get scaledjobs && kubectl get jobs'

Production Use

Build and push the controller image:

export IMG=<registry>/scaledjob-operator:<tag>
make docker-build docker-push IMG=$IMG

Deploy it:

make deploy IMG=$IMG

For an in-cluster Redis service, use a cluster DNS address in your ScaledJob:

redisAddress: redis.default.svc.cluster.local:6379

Do not use localhost in production unless Redis is running in the same container network as the controller. For normal Kubernetes deployments, localhost means the controller pod itself.

Observability

Status fields:

• status.queueDepth
• status.activeJobs
• status.desiredJobs
• status.lastScaleTime
• status.conditions

Events:

• RedisUnreachable
• QueueConnected
• CreatedJobs
• JobCreateFailed

Metrics:

• scaledjob_queue_depth
• scaledjob_active_jobs
• scaledjob_desired_jobs
• scaledjob_reconcile_total
• scaledjob_reconcile_errors_total

Development

Run tests:

make test

Regenerate manifests after RBAC or API marker changes:

make manifests generate

Clean up local demo resources:

kubectl delete -k config/samples/
kubectl delete deployment redis
kubectl delete service redis
kind delete cluster --name scaledjob-operator-dev