DTQ (Distributed Task Queue)

A lightweight Redis-backed task queue built in Python as a learning project toward a full distributed task system.

Repository Description

Building a simple distributed task queue in Python + Redis, with producers, workers, task serialization, and a registry-based execution model.

What It Does Today

• Defines serializable tasks with IDs, args/kwargs, and timestamps
• Pushes tasks to Redis lists from a producer
• Pulls tasks from Redis with blocking pop in a worker loop
• Registers callable task handlers and executes them by name
• Includes a local Docker Compose setup for Redis

Project Structure

• dtq/task.py: Task model and JSON serialization/deserialization
• dtq/queue.py: Redis-backed queue client (enqueue, dequeue, length, ping)
• dtq/worker.py: Task registry, worker loop, task execution
• tasks.py: Example registered task functions
• producer_main.py: Example producer script
• worker_main.py: Example worker runner
• docker-compose.yml: Local Redis service

Requirements

• Python 3.10+
• Docker + Docker Compose

Quick Start

1. Start Redis:

docker compose up -d

1. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate

1. Install dependencies:

pip install -r requirements.txt

1. Run a worker (terminal 1):

python worker_main.py

1. Enqueue demo tasks (terminal 2):

python producer_main.py

You should see the worker pick up and execute tasks from Redis.

Configuration

TaskQueue reads Redis connection settings from environment variables:

• REDIS_HOST (default: localhost)
• REDIS_PORT (default: 6379)

Example:

REDIS_HOST=localhost REDIS_PORT=6379 python worker_main.py

Next Milestones

If a worker thread dies mid-task (machine crash, OOM kill), that task is already gone from the Redis list - BRPOP removed it before we finished processing it. I have to fix this with the RPOPLPUSH reliability pattern - tasks only get removed from the queue after successful completion.