Order Service - Observability Testing System

A minimal, production-like Order API service focused on observability testing with Prometheus metrics, Grafana dashboards, and Loki for log aggregation.

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Order Service  │────▶│   Prometheus    │────▶│    Grafana      │
│   (FastAPI)     │     │   (Metrics)     │     │  (Dashboards)   │
│   Port: 8001    │     │   Port: 9090    │     │   Port: 3001    │
└────────┬────────┘     └─────────────────┘     └─────────────────┘
         │
         │ logs (stdout)
         ▼
┌─────────────────┐     ┌─────────────────┐
│    Promtail     │────▶│      Loki       │
│  (Log Shipper)  │     │ (Log Aggregator)│
│                 │     │   Port: 3100    │
└─────────────────┘     └─────────────────┘

Quick Start

Start All Services

docker-compose up --build

Service URLs

Service	URL	Credentials
Order Service API	http://localhost:8001	-
Prometheus	http://localhost:9090	-
Grafana	http://localhost:3001	admin / admin
Loki	http://localhost:3100	-

API Endpoints

Health Check

curl http://localhost:8001/api/health
# Response: {"status":"ok"}

Create Order

curl -X POST http://localhost:8001/api/orders \
  -H "Content-Type: application/json" \
  -d '{"item": "widget", "qty": 5}'
# Response: {"order_id":"uuid-here"}

Get Order

curl http://localhost:8001/api/orders/{order_id}
# Response: {"order_id":"...","item":"widget","qty":5,"created_at":"..."}

Prometheus Metrics

curl http://localhost:8001/metrics

Failure Injection

Method 1: FAIL Item

Create an order with item: "FAIL" to trigger HTTP 500:

curl -X POST http://localhost:8001/api/orders \
  -H "Content-Type: application/json" \
  -d '{"item": "FAIL", "qty": 1}'
# Response: HTTP 500 - Internal Server Error

Method 2: Admin Failure Rate

Set a percentage of requests to randomly fail:

# Set 30% failure rate
curl -X POST "http://localhost:8001/api/admin/failure?rate=0.3"

# Check current failure rate
curl http://localhost:8001/api/admin/failure

# Disable random failures
curl -X POST "http://localhost:8001/api/admin/failure?rate=0"

Observability Features

Structured Logging (JSON)

All logs are JSON formatted with:

• timestamp - ISO 8601 format
• level - INFO / ERROR
• service_name - "order-service"
• http_method - GET / POST
• http_route - Request path
• status_code - HTTP response code
• latency_ms - Request duration
• correlation_id - UUID per request
• error_message - Only for errors

Example log entry:

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "service_name": "order-service",
  "http_method": "POST",
  "http_route": "/api/orders",
  "status_code": 201,
  "latency_ms": 12.5,
  "correlation_id": "550e8400-e29b-41d4-a716-446655440000"
}

Prometheus Metrics

Available at /metrics:

• http_requests_total{method, path, status} - Total requests counter
• http_request_duration_seconds_bucket{method, path} - Latency histogram
• http_errors_total{method, path} - Error counter
• process_uptime_seconds - Service uptime gauge

Path Normalization: Dynamic IDs are normalized to {id} to prevent label explosion.

• /api/orders/550e8400-... → /api/orders/{id}

Example PromQL Queries:

# Total requests by path
sum(http_requests_total) by (path)

# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) * 100

# Request rate per second
sum(rate(http_requests_total[1m])) by (path)

# 95th percentile latency
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, path))

Verification Commands:

# Check metrics endpoint
curl http://localhost:8001/metrics | grep http_requests_total

# Verify in Prometheus UI
# Open http://localhost:9090 and query: http_requests_total

Alerting Rules

Prometheus alert rules in prometheus/alert_rules.yml:

• HighErrorRate - Fires when error rate > 5% over 1 minute
• HighErrorRateByStatus - Fires when 5xx rate > 5%
• OrderServiceDown - Fires when service is unreachable
• HighLatency - Fires when p95 latency > 1s

View alerts: http://localhost:9090/alerts

Grafana Dashboard

Pre-configured dashboard showing:

• Request Rate (by route)
• Error Rate (%)
• Latency (p50 & p95)
• Service Uptime
• Total Requests/Errors
• Service Status
• Recent Error Logs

Access: http://localhost:3001 (admin/admin)

Testing Scenarios

Generate Traffic

# Create multiple orders
for i in {1..10}; do
  curl -s -X POST http://localhost:8001/api/orders \
    -H "Content-Type: application/json" \
    -d "{\"item\": \"item-$i\", \"qty\": $i}"
done

Trigger High Error Rate Alert

# Set 50% failure rate
curl -X POST "http://localhost:8001/api/admin/failure?rate=0.5"

# Generate traffic (will have ~50% failures)
for i in {1..50}; do
  curl -s -X POST http://localhost:8001/api/orders \
    -H "Content-Type: application/json" \
    -d '{"item": "test", "qty": 1}'
done

# Check Prometheus alerts at http://localhost:9090/alerts
# Alert should fire within 1 minute

# Disable failures to resolve alert
curl -X POST "http://localhost:8001/api/admin/failure?rate=0"

Check Logs in Loki

Use Grafana's Explore feature with Loki datasource:

{job="containerlogs"} |= "order-service"

File Structure

├── docker-compose.yml          # All services orchestration
├── backend/
│   ├── Dockerfile              # Order service container
│   ├── server.py               # FastAPI application
│   └── requirements.txt        # Python dependencies
├── prometheus/
│   ├── prometheus.yml          # Prometheus config
│   └── alert_rules.yml         # Alert rules
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/        # Auto-provisioned datasources
│   │   └── dashboards/         # Dashboard provisioning
│   └── dashboards/
│       └── order-service.json  # Pre-built dashboard
├── loki/
│   └── loki-config.yml         # Loki configuration
└── promtail/
    └── promtail-config.yml     # Log shipping config

Stopping Services

docker-compose down

# Remove volumes (clean data)
docker-compose down -v

Observability Testing (Core Focus of This Project)

This project is intentionally designed to test observability signals in a production-like system, not just application functionality.

Unlike traditional API testing, the goal here is to validate that failures are observable, diagnosable, and actionable using logs, metrics, and alerts — the same signals used by SREs and on-call engineers in real systems.

The observability tests are implemented using Playwright and directly query Loki and Prometheus APIs, ensuring validation happens against the actual telemetry pipeline, not mocks.

---

Observability Test Coverage

1️⃣ Log Observability Test (Loki)

File: tests/observability/logs.spec.ts

What this test validates:

• A real application failure (POST /api/orders with item="FAIL")
• An ERROR log is emitted and ingested by Loki
• Logs are structured JSON, not plain text
• Required operational fields exist:
  • level = ERROR
  • http_route = /api/orders
  • status_code = 500
  • correlation_id is present

Why this matters:

This guarantees that when something breaks in production:

• Engineers can immediately understand what failed and why
• Logs are searchable, filterable, and traceable
• Correlation IDs enable request-level debugging

This test answers the operational question:

“Why did this request fail?”

---

2️⃣ Metric Observability Test (Prometheus)

File: tests/observability/metrics.spec.ts

What this test validates:

• HTTP request metrics are exposed at /metrics
• Prometheus scrapes the service successfully
• http_requests_total{method, path, status} increments correctly
• HTTP 5xx failures increment the metric with status="500"
• Tests handle eventual consistency by polling Prometheus until data appears

Why this matters:

Metrics are not useful unless they are:

• Accurate
• Stable (no label explosion)
• Reliable under real scrape timing

This test ensures metrics can be safely used for:

• Dashboards
• SLIs / SLOs
• Alerting

This test answers the operational question:

“How bad is the failure, and is it getting worse?”

---

3️⃣ Alert Validation Test (Prometheus Alerts)

File: tests/observability/alerts.spec.ts

What this test validates:

• Sustained failures trigger a real Prometheus alert
• Alert transitions correctly through states:

inactive → pending → firing → inactive

• Alert rules respect debounce windows (for: duration)
• Alerts resolve automatically once failures stop

Alert tested:

• HighErrorRateByStatus
  (Triggers when HTTP 5xx error rate exceeds 5% over 1 minute)

Why this matters:

This test validates operational readiness, not just data:

• On-call engineers are paged only when necessary
• Short blips do not cause false alerts
• Alerts self-heal when the system recovers

This test answers the operational question:

“Should a human be paged right now?”

---

Why This Testing Approach Is Different

Traditional testing answers:

• “Does the API return the right response?”

This project answers:

• “Is the system observable when it fails?”
• “Can operators diagnose issues quickly?”
• “Are alerts trustworthy?”

Key characteristics of this approach:

• ✅ Tests real telemetry pipelines (Loki + Prometheus)
• ✅ No mocks for logs, metrics, or alerts
• ✅ Handles async ingestion and scrape delays
• ✅ Treats observability as a contract, not an afterthought

---

▶️ Running Observability Tests

Ensure the full stack is running:

docker-compose up --build

Run tests individually:

# Log observability test
npx playwright test tests/observability/logs.spec.ts

# Metric observability test
npx playwright test tests/observability/metrics.spec.ts

# Alert validation test (takes ~2–3 minutes)
npx playwright test tests/observability/alerts.spec.ts