Reliable LLM Gateway Lab

A small reliability lab for LLM gateway behavior and serving-path failure modes.

The project simulates a gateway in front of model-serving backends. It includes an inline safeguard check, primary/fallback routing, timeout budgets, retry limits, circuit breaking, rate limiting, Prometheus metrics, and load testing.

It does not call paid LLM APIs. All model and safeguard services are mock FastAPI services so the lab is reproducible on a laptop.

Why This Exists

LLM serving reliability is different from ordinary API reliability. A single request can consume scarce model-serving capacity, hold memory for a long-running generation, depend on safety/safeguard checks, and stream output over time.

This lab makes those reliability controls visible:

• bounded backend calls with timeouts
• controlled retry behavior
• primary-to-fallback routing
• circuit breaking for repeated backend failures
• fail-closed safeguard behavior
• per-client rate limiting
• Prometheus metrics for latency, fallback, timeout, safeguard, TTFT, and generated tokens

Architecture

loadgen / curl client
  -> gateway:8000
      -> safeguard service:8003
      -> primary model backend:8001
      -> fallback model backend:8002
      -> Prometheus metrics at gateway:8000/metrics

The default gateway policy is:

1. Run safeguard check first.
2. Block if safeguard returns block or cannot return a clean allow.
3. Try the primary model backend.
4. Retry primary once for timeout or 5xx.
5. Fall back to the fallback backend if primary still fails.
6. Open the primary circuit after repeated failures.
7. Route directly to fallback while the primary circuit is open.

Quickstart

docker compose up --build

Health check:

curl http://localhost:8000/health

Send a request:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -H "X-Client-ID: demo" \
  -d '{"prompt":"Explain BGP route selection","max_tokens":50,"user_tier":"standard"}'

Inspect metrics:

curl http://localhost:8000/metrics

Failure Injection

Set the primary backend to timeout:

curl -X POST http://localhost:8001/mode \
  -H "Content-Type: application/json" \
  -d '{"mode":"timeout"}'

Send a request. It should fall back to the fallback backend:

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -H "X-Client-ID: fallback-demo" \
  -d '{"prompt":"test","max_tokens":50}'

Set primary back to normal:

curl -X POST http://localhost:8001/mode \
  -H "Content-Type: application/json" \
  -d '{"mode":"normal"}'

Set safeguard to block:

curl -X POST http://localhost:8003/mode \
  -H "Content-Type: application/json" \
  -d '{"mode":"block"}'

Reset gateway state:

curl -X POST http://localhost:8000/admin/reset

Load Test

Install locally, then run:

python -m venv .venv
source .venv/bin/activate
pip install -e .
python loadgen/run_load.py --url http://localhost:8000/chat --requests 100 --concurrency 10

Example output:

requests: 100
successes: 100
errors: 0
fallback count: 0
rate limited count: 0
p50 latency: 0.134s
p95 latency: 0.180s
p99 latency: 0.204s

Key Metrics

• llm_gateway_requests_total
• llm_gateway_request_duration_seconds
• llm_gateway_backend_requests_total
• llm_gateway_backend_latency_seconds
• llm_gateway_backend_timeouts_total
• llm_gateway_fallbacks_total
• llm_gateway_circuit_open_total
• llm_gateway_safeguard_requests_total
• llm_gateway_safeguard_latency_seconds
• llm_gateway_safeguard_blocks_total
• llm_gateway_rate_limited_total
• llm_gateway_simulated_ttft_seconds
• llm_gateway_tokens_generated_total

Key Learnings

• Gateway reliability is admission control for expensive model-serving infrastructure.
• Retries need budgets because uncontrolled retry traffic can amplify outages.
• Fallback improves availability, but without a circuit breaker it can keep hammering an unhealthy backend.
• Safeguard services are reliability-critical because they sit on the request path and affect both availability and safety behavior.
• Request count is not enough for LLM systems; token volume, TTFT, backend timeouts, safeguard latency, and fallback rate are better operational signals.
• A small lab can model production reliability concepts without pretending to operate production-scale GPUs.

Docs

• Architecture
• Failure Scenarios
• Learning Notes
• Primary Timeout Incident
• Safeguard Slow Incident