Atlas - AI Job Orchestration Platform

Production-grade asynchronous AI job orchestration infrastructure.

Overview

Atlas is designed to reliably execute AI inference jobs at scale. The infrastructure is the product; AI models are plugins.

Supported Job Types (V1):

• OCR
• Text Summarization
• Translation

Tech Stack

• API: FastAPI (Python 3.11+)
• Database: PostgreSQL 16 (async SQLAlchemy)
• Queue: Redis 7 + Celery 5.x
• Workers: Celery Workers (horizontal scaling)
• Logging: structlog (JSON format)
• Metrics: Prometheus

Quick Start

Prerequisites

• Docker + Docker Compose
• Python 3.11+ (for local development)

Development Setup

1. Clone the repository

   git clone <repository-url>
   cd atlas

2. Copy environment file

   cp .env.example .env

3. Start services with Docker Compose

   docker compose up -d postgres redis

4. Install Python dependencies

   pip install -r requirements.txt

5. Run database migrations

   alembic upgrade head

6. Start the API server

   uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

7. Start Celery workers (in separate terminal)

   celery -A app.worker.celery_app worker --loglevel=info --concurrency=4

Docker Compose (All Services)

docker compose up -d

This starts:

• API server (port 8000)
• 4 Celery workers
• PostgreSQL (port 5432)
• Redis (port 6379)

API Endpoints

Method	Endpoint	Description
POST	/api/v1/jobs	Submit a new job
GET	/api/v1/jobs/{id}	Get job status
GET	/api/v1/jobs	List jobs
GET	/api/v1/workers	List workers
GET	/health	Health check
GET	/metrics	Prometheus metrics

Example: Submit OCR Job

curl -X POST http://localhost:8000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "ocr",
    "payload": {"image_url": "https://example.com/image.png"}
  }'

Response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "QUEUED",
  "created_at": "2026-06-26T14:00:00Z"
}

Example: Poll Job Status

curl http://localhost:8000/api/v1/jobs/550e8400-e29b-41d4-a716-446655440000

Job Lifecycle

CREATED ? QUEUED ? RUNNING ? COMPLETED
                        +? FAILED ? RETRYING ? QUEUED
                                        +? DEAD_LETTER

Architecture

Testing

Run all tests

pytest

Run unit tests only

pytest tests/unit -v

Run integration tests only

pytest tests/integration -v

Run with coverage

pytest --cov=app --cov-report=html

Observability

Logs

Structured JSON logs to stdout. View with:

docker compose logs -f api
docker compose logs -f worker

Metrics

Prometheus metrics at GET /metrics:

• atlas_jobs_created_total
• atlas_jobs_completed_total
• atlas_jobs_failed_total
• atlas_queue_depth
• atlas_execution_time_seconds
• atlas_workers_active

Health Check

curl http://localhost:8000/health

Configuration

Variable	Default	Description
DATABASE_URL	postgresql+asyncpg://...	PostgreSQL connection
REDIS_URL	redis://redis:6379/0	Redis connection
CELERY_BROKER_URL	redis://redis:6379/0	Celery broker
WORKER_CONCURRENCY	4	Worker concurrency
WORKER_MAX_RETRIES	3	Default max retries
RATE_LIMIT_REQUESTS	100	Rate limit requests

Project Structure

atlas/
+-- app/
�   +-- domain/           # Business entities, enums
�   +-- application/      # Services, repositories
�   +-- infrastructure/   # DB, queue, external
�   +-- api/              # FastAPI routes, schemas
�   +-- models/           # Model handlers
�   +-- services/         # Business logic
�   +-- worker/           # Celery workers
�   +-- core/             # Config, logging, metrics
+-- tests/
�   +-- unit/
�   +-- integration/
+-- alembic/              # Database migrations
+-- docs/                 # Engineering docs
+-- docker-compose.yml

License

MIT