A distributed key-value Database-as-a-Service built from scratch. Primary-backup replication, automatic leader election, RPS-based autoscaling, and fast failure recovery — all observable through a live operator console.
Writes go through Nginx to a single elected leader. The leader applies the operation to its local LevelDB, publishes it to a RabbitMQ fanout for fast follower sync, and commits it to Kafka as a durable WAL. Reads are load-balanced across followers, which serve from their own LevelDB replica.
The orchestrator watches ZooKeeper for membership changes and handles everything that happens in the control plane: leader election when a leader is missing, spawning or trimming followers to match a target RPS, and pushing updated Nginx upstream configs when topology changes. It never touches the read/write request path.
When Nginx detects an upstream failure it calls back to the orchestrator, which probes the worker and — if dead — deletes its ZooKeeper znode immediately rather than waiting for session expiry. This cuts failover from ~30s to ~2s.
Topology graph + event stream during a leader failover
Leader crash → election → new leader promoted → Nginx config reconciled. Visible in the event log in real time.
Workload tab — live KPIs under load
p50/p95/p99 latency, reads/writes per second, error count, rolling time-series chart. Autoscaler spawning followers visible in the topology graph simultaneously.
┌─────────────────────────────────────────────────────────────────────────────┐
│ FRONTEND (React / Vite) │
│ - Observability tab: topology graph, metrics, scaling config, event log │
│ - Workload tab: workload config, live KPIs, time-series chart │
└──────────────────────┬──────────────────────────────────────┬───────────────┘
│ HTTP / WS │ HTTP
▼ ▼
┌────────────────────────────────────┐ ┌───────────────────────────────────┐
│ ORCHESTRATOR (FastAPI) │ │ WORKLOAD GENERATOR (FastAPI) │
│ - ZooKeeper children watch │ │ - Locust Environment + Runner │
│ - leader election │ │ - /api/jobs start / stop / stats │
│ - RPS-based autoscaling │ │ - runtime workload config │
│ - Nginx config source of truth │ │ - resets autoscaler on stop │
│ - upstream failure handling │ └──────────────────┬────────────────┘
│ - Prometheus HTTP SD │ │
└──────────────────────┬─────────────┘ │ HTTP traffic
│ POST /internal/reconcile ▼
│ (on topology change) ┌────────────────────────────┐
▼ │ target: nginx:80 │
┌─────────────────────────────────────────────────────────────────────────────┐
│ NGINX (data-plane router) port 80 │
│ /write → leader_upstream │
│ /read → follower_upstream (round-robin across followers, retry on fail) │
│ /api/stats → stub_status (used by orchestrator for autoscaling) │
│ │
│ RECONCILE SIDECAR (reconcile.py) port 8081 │
│ --bootstrap: pulls config from orchestrator on container start │
│ --serve: handles POST /internal/reconcile → fetch + write + reload │
└──────────────┬──────────────────────────────────────────────┬───────────────┘
│ writes │ reads (round-robin)
▼ ▼
┌──────────────────────────────┐ ┌────────────────────────────────────┐
│ LEADER WORKER │ │ FOLLOWER WORKER(S) │
│ - LevelDB writes │ │ - LevelDB reads │
│ - RabbitMQ fanout publish │─────────▶│ - RabbitMQ sync consumer │
│ - Kafka WAL publish │ │ - snapshot recovery on startup │
│ - periodic snapshots │ └────────────────────────────────────┘
└──────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ KAFKA + RABBITMQ + ZOOKEEPER │
│ - Kafka: durable write-ahead log (dbaas-writes WAL, dbaas.events stream) │
│ - RabbitMQ: replication fanout to followers (sync exchange) │
│ - ZooKeeper: worker membership, leader role, snapshot offsets │
└─────────────────────────────────────────────────────────────────────────────┘
| Layer | Technology |
|---|---|
| Control plane | Python, FastAPI, Kazoo (ZooKeeper), APScheduler |
| Data plane | Nginx, LevelDB (plyvel) |
| Replication | RabbitMQ (fanout), Kafka (WAL + replay) |
| Coordination | ZooKeeper |
| Load testing | Locust (embedded runner) |
| Frontend | React, Vite, Recharts |
| Observability | Prometheus, Grafana, OpenTelemetry / Tempo |
| Infrastructure | Docker, Docker Compose |
Leader election — when the orchestrator detects no leader in ZooKeeper, it ranks candidates by their last snapshot offset (most caught-up replica first), with proc_id as a deterministic tie-break. The elected node's znode is updated; the worker process picks up the change via DataWatch and swaps its subprocess from follower.py to leader.py.
Dual-channel replication — writes go to RabbitMQ fanout for low-latency follower propagation and to Kafka as a durable WAL. They serve different purposes: RabbitMQ minimises replica lag under normal operation; Kafka is what followers replay on startup. A follower that crashed and recovered locally skips the leader fetch entirely if its snapshot is fresh enough.
Nginx as active failure detector — Nginx intercepts upstream 5xx errors and calls back to the orchestrator via internal locations (@upstream_failure, @report_upstream_outcome). The orchestrator sends two health probes (750ms timeout, 150ms apart) before acting. On confirmed failure it deletes the ZooKeeper znode directly, which triggers election and Nginx reconcile without waiting for ZooKeeper session expiry.
Two autoscaling loops — an interval-based scheduler derives RPS from Nginx stub_status counter deltas and adjusts follower count to a target. A separate ZooKeeper ChildrenWatch handles crashes between scheduler ticks. They share a pending_spawns counter to avoid double-spawning during churn. Both scaling_interval_seconds and rps_per_worker are adjustable at runtime without restart.
Versioned Nginx config — the orchestrator renders Nginx config and hashes it (SHA-256 over sorted {leader, followers}). The reconcile sidecar skips the reload if the current version matches. This makes every topology event trigger a reconcile call without causing unnecessary Nginx reloads.
# Requires Docker Desktop with the default Docker socket enabled
docker compose -f docker-compose-local.yml up --buildThe cluster is ready when the orchestrator logs show "Scheduler started".
| Service | URL |
|---|---|
| Operator console | http://localhost:5173 |
| Orchestrator API | http://localhost:8080 |
| Nginx (data plane) | http://localhost:80 |
| Workload Generator | http://localhost:8089 |
| Grafana | http://localhost:3001 |
| Prometheus | http://localhost:9090 |
# Quick health check
curl http://localhost:8080/health
curl http://localhost:80/health
curl http://localhost:8089/healthTrigger a failover manually:
- Open the operator console → Observability tab
- Click a worker node → crash it from the Node Detail panel
- Watch the event log:
worker_crashing→leader_election_started→leader_elected→ Nginx reconcile
Run a load test and watch the autoscaler:
- Go to the Workload tab, set concurrent users and a key distribution (
zipfis interesting) - Start the job and watch followers spawn in the topology graph as RPS climbs
- Stop the job — autoscaler resets and trims back to 1 follower
Key environment variables (can also be set live via the API for autoscaling params):
| Variable | Default | Description |
|---|---|---|
SCALING_INTERVAL_SECONDS |
30 |
Autoscaler evaluation interval |
SCALING_RPS_PER_WORKER |
10.0 |
RPS target per follower |
SNAPSHOT_INTERVAL |
2000 |
Writes between leader snapshots |
FAILURE_PROBE_COOLDOWN_SEC |
1.5 |
Minimum time between probes of the same worker |
Detailed architecture, component internals, and design decisions are in _docs/system/: