A production-oriented speech inference platform built on faster-whisper. Demonstrates multi-GPU routing, queue-based request prioritization, API hardening, and operational safeguards for real-world transcription workloads.
- GPU-backed inference serving with heterogeneous hardware (RTX 4090 + RTX 3090)
- Weighted worker routing with explicit scheduling tradeoffs
- API-level concurrency controls and priority-based queuing
- SSRF-safe URL ingestion with validation pipeline
- Kubernetes-ready health probes and operational monitoring
- Security-first API key management (double-hashed, never stored in plaintext)
graph TB
Client -->|HTTPS| Traefik
Traefik --> Manager["Manager API :8000"]
Manager -->|weighted 60%| W1["Worker 1 — RTX 4090 :8001<br/>faster-whisper-large-v2"]
Manager -->|weighted 40%| W2["Worker 2 — RTX 3090 :8002<br/>faster-whisper-large-v3-turbo"]
Manager --> PG[(PostgreSQL)]
Manager --> Redis[(Redis<br/>3 Priority Queues)]
W1 --> Silero["Silero VAD · CPU"]
W2 --> Silero
The manager is the control plane: it validates requests, routes to GPU workers via weighted selection, and stores metrics. Workers are stateless inference executors that load Whisper models and run VAD preprocessing on CPU before GPU transcription.
- Multi-GPU weighted routing across heterogeneous GPUs (60/40 split)
- OpenAI-compatible
POST /v1/audio/transcriptionsendpoint - Silero VAD music edge detection and trimming (conservative/balanced/aggressive presets)
- 3-tier priority queues (high/medium/low) via Redis + RQ
- File upload + URL fetch with SSRF protection (private IP blocking, redirect validation, HTTPS-only)
- 5 output formats: JSON, verbose_json, plain text, SRT, VTT
- Double-hashed API keys (Blake2b + HMAC-SHA3-256)
- RFC 7807 error responses
- K8s-ready health probes (
/livez,/startupz,/readyz,/healthz) - Per-key concurrency limits + global semaphore
- Docker + Docker Compose
- NVIDIA GPU(s) with CUDA 12.6+ drivers
- NVIDIA Container Toolkit
cp .env.example .env
# Fill in all required values in .env (secrets, database credentials)
docker compose up --buildServices start on:
- Manager API:
http://localhost:8000 - Worker 1 (RTX 4090):
http://localhost:8001 - Worker 2 (RTX 3090):
http://localhost:8002
python -m venv .venv && source .venv/bin/activate
pip install -r manager_api/requirements.txt
# Requires PostgreSQL + Redis running locally
cd manager_api
uvicorn src.__main__:app --reload --port 8000curl -X POST http://localhost:8000/v1/audio/transcriptions \
-H "Authorization: Bearer $API_KEY" \
-F file=@audio.mp3 \
-F model=whisper-large-v3-turbo \
-F response_format=jsonResponse:
{
"text": "Hello, this is a transcription test.",
"duration": 5.12,
"processing_time": 0.83,
"speedup": 6.17,
"language": "en"
}curl -X POST http://localhost:8000/v1/audio/transcriptions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://cdn.example.com/audio.mp3",
"model": "whisper-large-v3-turbo",
"response_format": "verbose_json",
"strip_music_edges": true
}'Response (verbose_json):
{
"text": "Welcome to the show...",
"duration": 124.5,
"processing_time": 18.2,
"speedup": 6.84,
"language": "en",
"segments": [
{"id": 0, "start": 0.0, "end": 3.2, "text": "Welcome to the show."},
{"id": 1, "start": 3.2, "end": 7.8, "text": "Today we discuss..."}
]
}{
"type": "about:blank",
"title": "Validation Failed",
"status": 422,
"detail": "The request contains invalid parameters",
"instance": "/v1/audio/transcriptions",
"timestamp": "2025-01-20T15:30:00Z"
}Heterogeneous GPUs (4090 vs 3090) have different throughput characteristics. Static weights (60/40) are simpler and more predictable than dynamic load balancing --- the split reflects observed throughput ratios. Trade-off: doesn't adapt to burst patterns, but eliminates scheduling complexity and avoids the need for a health-polling feedback loop.
API keys carry a priority level, enabling different SLA expectations per consumer. Three Redis queues (high/medium/low) provide simple but effective differentiation --- workers drain high-priority first so latency-sensitive workloads get served faster. Trade-off: no preemption, queue-head blocking is possible under sustained load.
A global semaphore (10) prevents GPU saturation. A per-key limit (3) prevents a single consumer from monopolizing capacity. Simple but effective fairness without complex scheduling or admission control.
Podcast and audiobook content often has music intros/outros. VAD detects speech boundaries on CPU, then trims before GPU inference --- reducing wasted GPU compute on non-speech audio. Three sensitivity presets (conservative/balanced/aggressive) cover different content types without requiring per-request tuning.
Drop-in replacement for existing OpenAI Whisper API consumers. Reduces integration effort for clients and provides a familiar interface for developers.
- Weighted routing is configured statically --- reflects observed throughput, not live load
- Queues are priority-based, not deadline-based
- Manager is the control plane; workers are stateless inference executors
- URL fetching is bounded by configurable size/time limits and private-network restrictions
- First request per worker takes 5--10 minutes (model download from Hugging Face); health probes have a 600s startup window
| Scenario | Behavior |
|---|---|
| Worker unavailable | Manager returns 503; weighted selection does not retry |
| URL fetch fails | Timeout or validation error returned as RFC 7807 |
| Malformed audio | Worker returns transcription error; temp files cleaned up |
| Auth failure | 401/403 with RFC 7807 response |
| Queue backlog | Jobs queued in Redis; no backpressure beyond concurrency limits |
| Model cold start | First request takes 5--10 min; health probes have 600s startup window |
| GPU OOM | Recovery is process-level; no capacity-aware admission control |
- Single-host deployment --- no distributed scheduler, no cross-node routing
- Static weights --- routing is not load-aware; doesn't adapt to burst patterns
- No autoscaling --- fixed worker count per GPU
- Worker health is binary --- readiness probe, but no capacity-aware scheduling
- No persistent job store --- results are returned synchronously, not stored for later retrieval
- API keys:
whi.YYYYMM.{base64(32 bytes)}--- double-hashed (Blake2b + HMAC-SHA3-256), never stored in plaintext - URL fetch: SSRF-safe (private IP blocking, redirect validation, HTTPS-only, proxy env ignored)
- Auth: Bearer tokens for API/admin, Basic Auth for health checks
- All secrets are required env vars with no defaults in code
See SECURITY.md for the full security policy.
All configuration is via environment variables. See .env.example for the complete list with descriptions.
Key variables:
| Variable | Description |
|---|---|
POSTGRES_* |
Database connection |
REDIS_* |
Redis connection |
ADMIN_TOKEN |
Bearer token for admin endpoints |
API_KEY_SECRET |
HMAC secret for API key hashing |
API_KEY_PEPPER |
Salt for Blake2b hashing |
HEALTH_CHECK_USERNAME/PASSWORD |
Basic auth for /healthz |
NUM_WORKERS_4090/3090 |
Uvicorn worker count per GPU |
PRELOAD_MODEL_4090/3090 |
Whisper model to preload |
whisper-inference/
├── manager_api/
│ ├── src/
│ │ ├── __main__.py # FastAPI app, middleware, lifespan
│ │ ├── api/
│ │ │ ├── routers/
│ │ │ │ ├── transcribe.py # POST /v1/audio/transcriptions
│ │ │ │ ├── health_router.py # /livez, /readyz, /healthz
│ │ │ │ ├── api_keys.py # API key management
│ │ │ │ └── users.py # User management
│ │ │ └── dependencies/
│ │ │ └── api_key.py # API key validation
│ │ ├── config/settings.py # Pydantic Settings (env vars)
│ │ ├── models/ # SQLAlchemy models + Pydantic schemas
│ │ ├── repositories/ # Generic repository pattern
│ │ ├── middleware/ # RFC 7807 error handlers, logging
│ │ └── utils/
│ │ ├── crypto.py # API key generation + hashing
│ │ ├── url_fetch.py # SSRF-safe URL fetcher
│ │ └── storage.py # File upload handling
│ ├── alembic/ # Database migrations
│ ├── tests/ # Unit tests
│ └── manager.Dockerfile
├── worker_api/
│ ├── main.py # Worker: /transcribe + health probes
│ ├── vad_silero.py # Silero VAD integration
│ └── worker.Dockerfile # nvidia/cuda base image
├── docker-compose.yaml # Full orchestration
├── .env.example # Environment variable reference
└── docs/
├── ARCHITECTURE.md # Detailed architecture reference
└── openapi/ # OpenAPI 3.1.0 specification