diff --git a/docs/lola-rate-limiting.md b/docs/lola-rate-limiting.md index c722035..30e3e8b 100644 --- a/docs/lola-rate-limiting.md +++ b/docs/lola-rate-limiting.md @@ -1,625 +1,265 @@ -# LOLA Rate Limiting Implementation Guide +# LOLA Rate Limiting Implementation ## Overview This document provides a comprehensive guide to the rate limiting system implemented in the ActivityPub LOLA testbed. The rate limiting middleware ensures LOLA specification compliance while protecting OAuth and API endpoints from abuse. -**LOLA Compliance**: Per LOLA specification: *"The source server MAY rate limit requests by sending a 429 Too Many Requests response as defined in RFC6585, with a Retry-After header."* - ## Table of Contents 1. [LOLA Specification Requirements](#lola-specification-requirements) -2. [Implementation Architecture](#implementation-architecture) -3. [Rate Limit Configuration](#rate-limit-configuration) -4. [Request Processing Flow](#request-processing-flow) -5. [Real-World Usage Scenarios](#real-world-usage-scenarios) -6. [Development Testing](#development-testing) -7. [Production Readiness Assessment](#production-readiness-assessment) -8. [Federation Compatibility](#federation-compatibility) -9. [Monitoring and Troubleshooting](#monitoring-and-troubleshooting) +2. [Design Goals](#design-goals) +3. [Implementation Architecture](#implementation-architecture) +4. [Rate Limit Configuration](#rate-limit-configuration) +5. [Client IP Detection](#client-ip-detection) +6. [The 429 Response Contract](#the-429-response-contract) +7. [Cloud Run and Per-Instance Counters](#cloud-run-and-per-instance-counters) +9. [Development Testing](#development-testing) --- ## LOLA Specification Requirements -### RFC6585 Compliance +LOLA §6.7, *Load Management During Fetching of Content*: -Our implementation strictly follows [RFC6585 - Additional HTTP Status Codes](https://tools.ietf.org/html/rfc6585) for 429 responses: +> The source server **MAY** rate limit requests by sending a 429 Too Many Requests response as defined in [RFC6585], with a Retry-After header. If the destination receives a 429 response status code, it **SHOULD** respect the Retry-After header and resume its requests after the chosen delay. -- **Status Code 429**: "Too Many Requests" -- **Retry-After Header**: Specifies when client may retry (in seconds) -- **Response Body**: Human-readable explanation +Parsed carefully, this is a very light obligation on a source server: -### ActivityPub Federation Requirements +| Dimension | Obligation | +|---|---| +| Rate limit at all? | **Optional** — MAY. | +| Status code | `429` — required | +| `Retry-After` header | Required as part of the described mechanism | +| Explanatory body | SHOULD, per RFC6585 | +| Counting accuracy | **Unspecified** | +| Global consistency | **Unspecified** | +| Algorithm / limits / identity key | **Our choice** | -Rate limiting is crucial for LOLA source servers because: +RFC6585 is explicit that it "does not define how the origin server identifies the user, nor how it counts requests." So the algorithm, the limit values, the identity key, the window and the consistency model are all implementation choices, not compliance requirements. -1. **External Server Protection**: Destination servers implementing LOLA need reliable source servers -2. **OAuth Endpoint Security**: OAuth authorization is critical for account portability workflows -3. **Service Availability**: Prevents individual clients from impacting legitimate LOLA operations -4. **Specification Compliance**: Enables real-world LOLA implementations to test against standards-compliant infrastructure +**The normative weight sits on the destination.** The only SHOULD in §6.7 is about destinations honoring `Retry-After`. That is what destination developers must implement — and this testbed exists so they can implement and verify it against a compliant source. -### LOLA-Specific Considerations +--- -- **Account Migration Workflows**: OAuth endpoints get stricter limits due to sensitivity -- **Federation Compatibility**: CORS headers enable destination servers to handle rate limits gracefully -- **Educational Balance**: Limits are strict enough for protection but permissive enough for learning +## Design Goals ---- +Because compliance is nearly free here, the design optimizes for something else: **usefulness to destination implementers.** -## Implementation Architecture +1. **Predictable, correlatable 429s.** A destination developer must be able to trigger a 429, read `Retry-After`, back off, retry, and confirm their logic works. A limiter that fires for reasons unrelated to the caller's own request pattern is worse than no limiter, because backoff logic cannot be developed against an arbitrary signal. +2. **A machine-readable body.** The 429 uses the same JSON error contract as every other endpoint, so it can be parsed rather than pattern-matched. +3. **Light abuse dampening.** Secondary. This is a testbed, not an enforcement boundary. -### Sliding Window Algorithm +Accuracy of counting is explicitly *not* a goal — see Cloud Run and per-instance counters. -The rate limiting uses a **sliding time window** approach that tracks request timestamps per client IP: +--- -```python -# Example for IP 203.0.113.42 with 5-minute window -request_timestamps = [ - 1693315800, # 14:30:00 - 1693315815, # 14:30:15 - 1693315930, # 14:32:10 - 1693316025 # 14:33:45 -] +## Implementation Architecture -# At 14:34:02, window is 14:29:02 to 14:34:02 -# All 4 requests are within window -``` +Implemented in `testbed/core/middleware/rate_limiting.py` as `RateLimitingMiddleware`, registered in `settings/base.py` early in the `MIDDLEWARE` list. -**Algorithm Benefits:** -- **Fair Distribution**: Allows bursts but prevents sustained abuse -- **Automatic Recovery**: Old requests automatically expire from window -- **Memory Efficient**: Only stores timestamps, not full request data +### Per-(rule, client) fixed-window counters -### In-Memory Storage Design +Each request resolves to exactly one **rule** by longest matching path prefix. Counting happens against a bucket keyed by that rule *and* the client: -**Data Structure:** -```python -# Collections.defaultdict(list) stores IP -> timestamp list -request_counts = { - '203.0.113.42': [1693315800, 1693315815, 1693315930], - '198.51.100.10': [1693316000, 1693316050], - # ... more IPs -} ``` - -**Storage Characteristics:** -- **Temporary**: Lost on server restart (acceptable for basic production) -- **Per-Server**: Each application instance has separate counters -- **Bounded**: Automatic cleanup prevents unlimited growth -- **Fast**: In-memory access with O(1) IP lookup - -### Client IP Detection - -The middleware attempts to identify real client IPs through proxy headers: - -```python -def get_client_ip(self, request): - # Priority order for IP detection - - # 1. X-Forwarded-For (most common proxy header) - x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') - if x_forwarded_for: - return x_forwarded_for.split(',')[0].strip() - - # 2. X-Real-IP (alternative proxy header) - x_real_ip = request.META.get('HTTP_X_REAL_IP') - if x_real_ip: - return x_real_ip - - # 3. Direct connection (fallback) - return request.META.get('REMOTE_ADDR', '127.0.0.1') +ratelimit:: ``` -**Production Considerations:** -- Configure proxy headers based on infrastructure -- Validate proxy sources to prevent header spoofing -- Consider subnet-based limiting for complex networks - -### Memory Management and Cleanup - -**Automatic Cleanup Process:** -- **Trigger**: Runs on every request -- **Age Limit**: Removes entries older than 1 hour -- **Scope**: Cleans both old timestamps and empty IP records -- **Performance**: O(n) where n = number of tracked IPs - -```python -def cleanup_old_entries(self, current_time, max_age=3600): - cutoff_time = current_time - max_age - - for ip in list(self.request_counts.keys()): - # Remove old timestamps - self.request_counts[ip] = [ - t for t in self.request_counts[ip] if t > cutoff_time - ] - - # Remove empty IP records - if not self.request_counts[ip]: - del self.request_counts[ip] -``` +Keying by rule is what keeps budgets isolated. Traffic to `/api/actors/` can never consume the `/oauth/authorize/` allowance. ---- +Each bucket is a single integer stored with a TTL equal to the rule's window: -## Rate Limit Configuration +- The first request of a window creates it with `cache.add(key, 1, window)` +- Later requests use `cache.incr(key)` +- Expiry is the cache's responsibility, so there is no cleanup pass and no unbounded growth -### Endpoint-Specific Limits - -The middleware applies different limits based on endpoint sensitivity: - -| Endpoint | Requests | Window | Rationale | -|----------|----------|---------|-----------| -| `/oauth/authorize/` | 10 | 5 minutes | Critical for LOLA authentication | -| `/oauth/token/` | 20 | 5 minutes | Token exchange endpoint | -| `/.well-known/oauth-authorization-server` | 30 | 1 minute | LOLA discovery | -| `/api/actors/` | 100 | 1 minute | LOLA collections | -| **Default** | 200 | 1 minute | General endpoints | - -### Path-Based Matching Logic - -The system uses **longest prefix matching** to find the most specific rate limit: - -```python -def get_rate_limit_for_path(self, path): - # Example: path = "/api/actors/123/followers" - # Matches: "/api/actors/" (not "/api/" or "/") - - best_match = self.default_limit - best_match_length = 0 - - for pattern, limit in self.rate_limits.items(): - if path.startswith(pattern) and len(pattern) > best_match_length: - best_match = limit - best_match_length = len(pattern) - - return best_match -``` +A companion `:reset` entry records when the window ends, so `Retry-After` reports the true remaining time rather than a whole window. -### Customization Options +### Accepted trade-off: fixed vs sliding window -**BasicRateLimitingMiddleware** (Development/Basic Production): -```python -rate_limits = { - '/oauth/authorize/': {'requests': 10, 'window': 300}, - '/oauth/token/': {'requests': 20, 'window': 300}, - '/api/actors/': {'requests': 100, 'window': 60}, -} -default_limit = {'requests': 200, 'window': 60} -``` +A fixed window permits up to 2× the limit across a boundary — N requests just before it, N just after. A sliding window is more precise. -**LOLARateLimitingMiddleware** (Strict Production): -```python -rate_limits = { - '/oauth/authorize/': {'requests': 5, 'window': 300}, # Stricter - '/oauth/token/': {'requests': 10, 'window': 300}, # Stricter - '/api/actors/': {'requests': 50, 'window': 60}, # Stricter -} -default_limit = {'requests': 100, 'window': 60} # More conservative -``` +For a dampening signal under a MAY, that imprecision is irrelevant and the simplicity is worth it. This is a deliberate choice, not an oversight. ---- +### Backend constraint -## Request Processing Flow +The design relies on `incr()` **preserving the key's TTL**, so that continued traffic cannot extend a window. If hammering while blocked pushed the window out, `Retry-After` would become a lie — and §6.7 asks destinations to trust that value. -### Step-by-Step Processing +`LocMemCache`, the default and what this deployment uses, preserves it: `incr()` writes straight to its internal dict under a lock and never touches the expiry table. Django's **database cache backend does not**. It inherits `BaseCache.incr`, which does `get()` then `set()` without a timeout, resetting the TTL to `DEFAULT_TIMEOUT`. Swapping to it would quietly turn this into a sliding window, and no test would fail — so do not change the cache backend without first checking that its `incr()` leaves the TTL alone. -Every HTTP request goes through this detailed evaluation: +### Fail-open -``` -1. REQUEST INTERCEPTION - ├── Middleware activates early in Django stack - ├── Positioned after sessions, before CSRF - └── Has access to session data for IP tracking - -2. CLIENT IDENTIFICATION - ├── Extract IP from proxy headers or direct connection - ├── Handle X-Forwarded-For, X-Real-IP scenarios - └── Fallback to REMOTE_ADDR if needed - -3. MEMORY CLEANUP - ├── Remove timestamps older than 1 hour - ├── Delete IPs with no recent activity - └── Maintain bounded memory usage - -4. RATE LIMIT EVALUATION - ├── Find most specific rate limit for request path - ├── Get request history for client IP - ├── Filter to requests within time window - ├── Count recent requests vs limit - └── Calculate retry time if exceeded - -5. DECISION & RESPONSE - ├── If under limit: Record request, continue - └── If over limit: Generate 429 response -``` +If the cache raises for any reason, the middleware logs and allows the request. A limiter is a dampening signal under a MAY; it must never take down the endpoints it protects. -### 429 Response Generation +--- -When rate limits are exceeded, the middleware generates RFC6585-compliant responses: +## Rate Limit Configuration -```http -HTTP/1.1 429 Too Many Requests -Content-Type: text/plain -Retry-After: 238 -Access-Control-Allow-Origin: * -Access-Control-Expose-Headers: Retry-After +All values live in `settings/base.py`, which every environment module inherits. The middleware keeps no copies of the tuned values — identical numbers in two files drift, and settings is where an operator looks to change them. -Rate limit exceeded. Try again in 238 seconds. -``` +| Setting | If absent | Purpose | +|---|---|---| +| `RATE_LIMIT_ENABLED` | defaults to `True` | Master switch | +| `RATE_LIMIT_RULES` | defaults to `[]` | Per-path rules; longest prefix wins | +| `RATE_LIMIT_DEFAULT` | **required** — raises | Fallback rule for unmatched paths | +| `RATE_LIMIT_EXEMPT_PREFIXES` | **required** — raises | Paths that never count | +| `RATE_LIMIT_TRUSTED_PROXY_DEPTH` | defaults to `0` | Trusted trailing `X-Forwarded-For` entries | -**Retry-After Calculation:** -```python -# Find when oldest request in window expires -oldest_request = min(recent_requests) -window_expires = oldest_request + rate_limit['window'] -retry_after = max(window_expires - current_time, 1) # At least 1 second -``` +The two inline defaults encode a **safety posture** rather than configuration: limiting stays on, and `X-Forwarded-For` stays untrusted, unless a deployment says otherwise. The two required settings deliberately have no fallback, so a missing value fails loudly instead of silently applying a number hidden in code. ---- +### Default rules -## Real-World Usage Scenarios +| Rule | Prefix | Limit | Window | +|---|---|---|---| +| `oauth_authorize` | `/oauth/authorize/` | 60 | 300s | +| `oauth_token` | `/oauth/token/` | 120 | 300s | +| `lola_discovery` | `/.well-known/oauth-authorization-server` | 60 | 60s | +| `lola_api` | `/api/actors/` | 120 | 60s | +| *(fallback)* | everything else | 300 | 60s | -### Scenario: External ActivityPub Server Testing LOLA +Limits are deliberately generous. One interactive OAuth authorization spans several requests — consent page, approval POST, redirect, token exchange — and this testbed exists for people to exercise that flow repeatedly. A limit that throttles honest integration testing would defeat its purpose. -**Background**: MastodonPlus.social is implementing LOLA account portability and testing their destination server against our testbed. +Longest-prefix matching means rules can be declared in any order; specificity decides. -**Timeline with Real Requests:** +### Per-environment behaviour -**14:30:00 - Developer starts OAuth testing** -``` -Request 1: POST /oauth/authorize/ from 203.0.113.42 -✅ ALLOWED (1/10 requests in window) -Response: 200 OK -Memory: ['14:30:00'] -``` +| Environment | Enabled | Notes | +|---|---|---| +| Development | **No** | `DEBUG=True` serves static through Django, so a page load would spend a dozen requests of budget. Set `DJANGO_RATE_LIMIT_ENABLED=1` to exercise it. | +| Test | **No** | Prevents unrelated suites being throttled. `test_rate_limiting.py` re-enables explicitly via `override_settings`. | +| CI | Yes | Inherits base defaults | +| Staging / Production | Yes | Plus `RATE_LIMIT_TRUSTED_PROXY_DEPTH=1` | -**14:30:15 to 14:33:45 - Rapid development testing** -``` -Requests 2-10: POST /oauth/authorize/ from 203.0.113.42 -✅ ALL ALLOWED (10/10 requests in window) -Memory: ['14:30:00', '14:30:15', ..., '14:33:45'] -``` +### Exemptions -**14:34:02 - Rate limit triggered** -``` -Request 11: POST /oauth/authorize/ from 203.0.113.42 -❌ RATE LIMITED! +Static and media matter only in development, where Django serves them; in staging and production they come from Cloud Storage and never reach this middleware. Health checks belong to Cloud Run rather than to any user and must not exhaust a user's allowance. -Calculation: -- Window: 14:29:02 to 14:34:02 (5 minutes) -- Recent requests: 10 (all within window) -- Limit: 10 per 5 minutes → EXCEEDED -- Oldest request: 14:30:00 -- Retry after: (14:30:00 + 300) - 14:34:02 = 238 seconds +--- -Response: HTTP 429 with Retry-After: 238 -``` +## Client IP Detection -**14:38:05 - Automatic recovery** -``` -Request 12: POST /oauth/authorize/ from 203.0.113.42 -✅ ALLOWED! +`X-Forwarded-For` is **appended to** by each proxy in a chain, so its leftmost entry is whatever the client sent and is entirely under the client's control. Reading index 0 lets a client mint a fresh bucket per request simply by varying the header. -Why? Sliding window moved: -- Window: 14:33:05 to 14:38:05 -- Oldest request (14:30:00) expired from window -- Now only 9 requests in current window -``` +`RATE_LIMIT_TRUSTED_PROXY_DEPTH` is the number of proxies that append to the header in front of this application. The real client sits that many entries in **from the right**, so anything injected by a client lands further left and is ignored. -### Memory State Evolution - -**Before Rate Limit (14:34:02):** -```python -request_counts = { - '203.0.113.42': [ - 1693315800, # 14:30:00 - 1693315815, # 14:30:15 - 1693315882, # 14:31:22 - 1693315930, # 14:32:10 - 1693315965, # 14:32:45 - 1693316000, # 14:33:20 - 1693316025, # 14:33:45 - # ... 3 more timestamps ... - ] - # Total: 10 requests in memory -} ``` +depth = 1, well-formed: "203.0.113.5, 130.211.0.1" + ^^^^^^^^^^^ client (index -2) -**After Rate Limit + Recovery (14:38:05):** -```python -request_counts = { - '203.0.113.42': [ - # 14:30:00, 14:30:15 expired (outside window) - 1693315882, # 14:31:22 - 1693315930, # 14:32:10 - # ... remaining requests ... - 1693316285 # 14:38:05 (new allowed request) - ] - # Total: 9 requests (oldest expired naturally) -} +depth = 1, client injects: "1.1.1.1, 203.0.113.5, 130.211.0.1" + ^^^^^^^^^^^ still the client ``` ---- +**Default is 0**, meaning `X-Forwarded-For` is not trusted at all and `REMOTE_ADDR` is used. Every deployment opts in by declaring how deep its own chain is. -## Development Testing +If the observed chain is shorter than the configured depth, the middleware logs a warning and falls back to `REMOTE_ADDR`, since the header is not the shape the deployment expects. -### Method 1: Browser Testing (Easiest) +The warning only fires when the chain is *shorter* than configured. A depth set too **high** fails silently, and lets a caller mint a fresh bucket per request by padding the header. -**OAuth Authorization Endpoint (10 requests/5 minutes):** +> **Verify before relying on it.** Production sets `1`, but that is a considered bet rather than a documented fact: Google specifies the `, ` format for external Application Load Balancers, which this service does not use, and documents no stable layout for the `run.app` / domain-mapping path. Send one request with no `X-Forwarded-For` and read `client resolution: xff_entries=N` from the logs, then set the depth to `N - 1`. Check on production via the custom domain — staging has no custom domain, so it cannot exercise that path. Correctable via `DJANGO_RATE_LIMIT_TRUSTED_PROXY_DEPTH` without a redeploy. -1. Start development server: `python manage.py runserver` -2. Navigate to OAuth URL: - ``` - http://127.0.0.1:8000/oauth/authorize/?client_id=YOUR_CLIENT_ID&response_type=code&scope=activitypub_account_portability - ``` -3. **Rapid refresh test**: Press F5 or Ctrl+R rapidly 11+ times -4. **Expected result**: 11th request shows "Rate limit exceeded" +--- -### Method 2: curl Command Testing (Most Reliable) +## The 429 Response Contract -**Quick Rate Limit Test:** -```bash -# Test OAuth authorization endpoint -for i in {1..11}; do - echo "Request $i:" - curl -v http://127.0.0.1:8000/oauth/authorize/ 2>&1 | grep -E "(HTTP|Retry-After)" - echo "---" -done -``` +Built by `build_rate_limit_error` in `testbed/core/utils/errors.py`. The whole response is the template — status, body and headers. -**Expected Output (11th request):** -``` -Request 11: -< HTTP/1.1 429 Too Many Requests -< Retry-After: 287 -< Access-Control-Allow-Origin: * ---- +```http +HTTP/1.1 429 Too Many Requests +Content-Type: application/json +Retry-After: 47 +Cache-Control: no-store +Access-Control-Allow-Origin: * +Access-Control-Expose-Headers: Retry-After ``` -### Method 3: Python Test Script (Most Controlled) - -```python -import requests -import time - -def test_oauth_rate_limiting(): - # Test OAuth endpoint rate limiting with detailed output - url = "http://127.0.0.1:8000/oauth/authorize/" - - print("Testing OAuth Rate Limiting (10 requests/5 minutes)") - print("=" * 50) - - for i in range(1, 12): - start_time = time.time() - response = requests.get(url) - end_time = time.time() - - print(f"Request {i:2d}: Status {response.status_code} " - f"({end_time - start_time:.3f}s)") - - if response.status_code == 429: - retry_after = response.headers.get('Retry-After', 'Not set') - cors_origin = response.headers.get('Access-Control-Allow-Origin', 'Not set') - - print(f" ⚠️ Rate Limited!") - print(f" 📅 Retry-After: {retry_after} seconds") - print(f" 🌐 CORS Origin: {cors_origin}") - print(f" 📝 Body: {response.text[:50]}...") - break - else: - print(f" ✅ Allowed") - - time.sleep(0.1) # Small delay between requests - -if __name__ == "__main__": - test_oauth_rate_limiting() +```json +{ + "error_code": "rate_limit_exceeded", + "detail": "Request rate limit exceeded", + "timestamp": "2026-07-27T14:33:05.123456+00:00", + "hint": "Too many requests: the limit is 60 per 300 seconds. Please wait 47 seconds before retrying", + "remediation": "Honor the Retry-After header, then resume with exponential backoff", + "endpoint": "/oauth/authorize/", + "method": "GET", + "request_id": "5f3c2e91-..." +} ``` -### Method 4: Testing Different Endpoints - -**Fastest to Trigger (Discovery - 30/minute):** -```bash -for i in {1..31}; do - echo -n "Request $i: " - curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/.well-known/oauth-authorization-server - echo -done -``` +Why each header is present: -**API Endpoints (100/minute):** -```bash -for i in {1..101}; do - curl -s -o /dev/null -w "Request $i: %{http_code}\n" http://127.0.0.1:8000/api/actors/1/ -done -``` +- **`Retry-After`** — §6.7 describes rate limiting as a 429 "with a Retry-After header"; destinations SHOULD honor it. +- **`Cache-Control: no-store`** — 429 is not in HTTP's heuristically-cacheable set, so a compliant cache would not store it anyway. Stating it explicitly is cheap defence against a non-compliant intermediary replaying a stale 429 at a client that has already backed off. +- **`Access-Control-Expose-Headers`** — ActivityPub federation clients are frequently browser-based. `Retry-After` is unreadable to `fetch()` unless explicitly exposed, which would leave a destination unable to honor the SHOULD above. -## Production Readiness Assessment - -### ✅ Production-Ready Aspects - -**LOLA Specification Compliance:** -- RFC6585 compliant 429 responses -- Proper Retry-After header calculation -- ActivityPub federation CORS support -- Standard HTTP semantics for automated clients - -**Security Protection:** -- OAuth endpoint protection against brute force -- Resource conservation preventing server overload -- Graceful degradation with clear error messaging -- IP-based tracking prevents individual client abuse - -**Operational Stability:** -- Automatic memory cleanup prevents leaks -- Reasonable limits balance protection with usability -- Basic proxy support for common nginx setups -- No external dependencies simplifying deployment - -### Production Deployment Scenarios - -**✅ Works Well For:** - -**Single-Server Deployments:** -- Small to medium LOLA testbeds (< 1000 users) -- Educational/research instances -- Single Django server behind nginx reverse proxy -- Community demonstration servers -- Docker containerized deployments (single instance) - -**Moderate Traffic Volumes:** -- Hundreds of OAuth requests per hour -- Developer testing and integration scenarios -- Community learning and experimentation -- Basic ActivityPub federation testing - -**Simple Infrastructure:** -- nginx → Django application server -- Cloud Run single instance deployments -- Basic load balancer with session affinity -- Development/staging environments - -### ⚠️ Current Limitations - -**Multi-Server Challenges:** -- **Issue**: Each app server maintains separate rate limit counters -- **Impact**: Rate limiting becomes less effective with horizontal scaling -- **Workaround**: Use sticky sessions or single-server deployments -- **Future**: Upgrade to Redis storage for shared state - -**Server Restart Behavior:** -- **Issue**: Rate limit memory cleared on application restart -- **Impact**: Brief period where all rate limits reset -- **Mitigation**: Usually not critical for basic production use -- **Future**: Persistent storage will maintain state - -**Advanced IP Detection:** -- **Issue**: Basic proxy header parsing may not handle complex chains -- **Impact**: Could rate limit wrong IPs in edge cases -- **Mitigation**: Configure X-Forwarded-For properly for your proxy setup -- **Future**: Enhanced proxy validation and trusted proxy lists - -### 🚀 When to Upgrade - -**Consider advanced rate limiting when you reach:** -- **Multiple Application Servers**: Need shared rate limit state -- **High Traffic Volumes**: Thousands of requests per hour requiring optimization -- **Complex Proxy Infrastructure**: Multiple load balancers, CDN, cloud proxies -- **Enterprise Security**: Need IP whitelisting, progressive penalties -- **Detailed Monitoring**: Require metrics, alerts, dashboards +The body uses the same `build_error_payload` as every other error in the project, so clients see one contract regardless of which layer rejected them. --- -## Federation Compatibility +## Cloud Run and Per-Instance Counters -### CORS Headers for ActivityPub Clients +Counters live in Django's cache. Under the default `LocMemCache` that is **per-process**, and Cloud Run autoscales. The effective ceiling is therefore: -Our rate limiting includes ActivityPub-specific CORS headers: - -```python -response['Access-Control-Allow-Origin'] = '*' -response['Access-Control-Expose-Headers'] = 'Retry-After' +``` +limit × gunicorn_workers × running_cloud_run_instances ``` -**Why This Matters:** -- External ActivityPub servers can read rate limit headers from JavaScript -- Enables destination servers to implement proper backoff logic -- Follows web standards for cross-origin resource sharing +Threads do not multiply it — they share one process's memory, which is exactly why `incr()` has to be atomic. Worker processes and Cloud Run instances do. The container currently runs `--workers 1`, so today this reduces to `limit × instances`. -### Error Response Format +Two consequences, stated plainly: -Our 429 responses follow standards that work with automated systems: +**These limits are best-effort dampening, not an enforcement boundary.** §6.7's MAY is what makes that acceptable rather than a compliance gap. Nothing in LOLA or RFC6585 asks for globally consistent counting. -```http -HTTP/1.1 429 Too Many Requests -Content-Type: text/plain -Retry-After: 180 -Access-Control-Allow-Origin: * -Access-Control-Expose-Headers: Retry-After -Date: Thu, 29 Aug 2025 19:34:02 GMT +**The multiplier has no stated upper bound, deliberately.** No `--max-instances` is configured on the Cloud Run service, so the platform default applies. Capping instances was considered and **declined**. -Rate limit exceeded. Try again in 180 seconds. -``` +**In practice the testbed usually runs a single instance**, where counters are effectively global. The multiplier only appears under concurrent load — which is also the situation where letting some extra traffic through matters least. + +Other platform interactions worth knowing: -**Machine-Readable Elements:** -- **Status Code 429**: Universally recognized -- **Retry-After Header**: Precise retry timing -- **CORS Headers**: Enable browser-based clients -- **Plain Text Body**: Human-readable explanation +- **Cold starts reset counters.** A scale-to-zero followed by a new instance starts every bucket empty. Acceptable under a MAY. +- **Static files never reach this middleware in staging/production**, because they are served from Cloud Storage. They only count in development, where `DEBUG=True` makes Django serve them. +- **Health checks are exempt** so platform probes do not consume user budget. --- -## Monitoring and Troubleshooting +## Development Testing -### Log Messages and Levels +Coverage lives in `testbed/core/tests/test_rate_limiting.py` (18 tests). Rate limiting is disabled by default in the test settings, so each test enables it explicitly with its own small rule set rather than depending on production limits. -**Warning Level (Always Logged):** -``` -WARNING Rate limit exceeded for IP 203.0.113.42 on /oauth/authorize/. Retry after 240 seconds. -``` +Areas covered: -**Info Level (LOLA-Specific Middleware):** -``` -INFO LOLA rate limit triggered: IP=203.0.113.42, path=/oauth/authorize/, retry_after=240s -``` +- **Bucket isolation** — traffic on one rule never drains another, and one client never affects another +- **Limit boundary** — N allowed, N+1 rejected, rejected requests never reach the view +- **Window semantics** — hammering while blocked does not extend `Retry-After` +- **429 contract** — JSON body keys, `Retry-After` range, cache and CORS headers +- **Client identification** — depth-0 ignores `X-Forwarded-For`; depth-1 counts in from the right and ignores injected entries; short chains fall back to `REMOTE_ADDR` +- **Exemptions** — static and health paths never count +- **Operational safety** — disabled switch, and fail-open on cache failure +- **Wiring** — one integration test through the real client proving the middleware is installed in `MIDDLEWARE` -**Debug Level (Development Only):** -``` -DEBUG Rate limit check: IP=127.0.0.1, path=/api/actors/1/, count=3/100, window=60s -``` +### Exercising it by hand -### Common Issues and Solutions +```bash +# Enable locally, then hammer an endpoint +DJANGO_RATE_LIMIT_ENABLED=1 python manage.py runserver -**Issue: Rate limits too strict for development** -```python -# Solution: Use BasicRateLimitingMiddleware instead of LOLARateLimitingMiddleware -MIDDLEWARE = [ - # ... - 'testbed.core.middleware.rate_limiting.BasicRateLimitingMiddleware', - # ... -] -``` +# Watch the limit engage +for i in $(seq 1 70); do + curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:8000/oauth/authorize/ +done +echo -**Issue: Rate limits not working after server restart** -``` -# Expected behavior: In-memory storage is cleared on restart -# This is normal for current implementation -# Solution: Wait for limits to rebuild naturally, or upgrade to persistent storage +# Inspect the 429 body and headers +curl -i http://127.0.0.1:8000/oauth/authorize/ ``` --- -## Configuration Reference - -### Middleware Setup - -**settings/base.py:** -```python -MIDDLEWARE = [ - 'django.middleware.security.SecurityMiddleware', - 'django.contrib.sessions.middleware.SessionMiddleware', - 'django.middleware.common.CommonMiddleware', - # Position rate limiting early to protect all endpoints - 'testbed.core.middleware.rate_limiting.BasicRateLimitingMiddleware', - 'django.middleware.csrf.CsrfViewMiddleware', - # ... rest of middleware stack -] -``` - -### Rate Limit Customization - -**Custom Rate Limits:** -```python -# In middleware class -rate_limits = { - '/oauth/authorize/': {'requests': 5, 'window': 300}, # 5 per 5 minutes - '/oauth/token/': {'requests': 15, 'window': 300}, # 15 per 5 minutes - '/api/actors/': {'requests': 50, 'window': 60}, # 50 per minute - '/.well-known/': {'requests': 20, 'window': 60}, # 20 per minute -} -``` - -## Conclusion - -The LOLA rate limiting implementation provides a solid foundation for protecting ActivityPub OAuth infrastructure while maintaining LOLA specification compliance. It balances security, usability, and educational value, making it suitable for development, demonstration, and basic production deployments. +### Log messages -The middleware's design enables easy enhancement for more advanced production scenarios while providing immediate value for LOLA compliance and basic abuse protection. +| Level | Message | Meaning | +|---|---|---| +| `INFO` | `Rate limit client resolution: xff_entries=… configured_depth=… resolved=…` | Emitted once per process on the first counted request. The chain shape this deployment actually sees — how you check the proxy depth without forcing a 429 | +| `WARNING` | `Rate limit exceeded ip=… rule=… path=…` | A request was rejected; includes limit, window and retry-after | +| `WARNING` | `X-Forwarded-For shorter than RATE_LIMIT_TRUSTED_PROXY_DEPTH` | Depth set too high; fell back to `REMOTE_ADDR`. Note the reverse case — depth too low — produces no warning | +| `ERROR` | `Rate limiting check failed, allowing request` | Cache failure; request was allowed (fail-open) | diff --git a/testbed/core/middleware/rate_limiting.py b/testbed/core/middleware/rate_limiting.py index 57ba1c3..28ff95b 100644 --- a/testbed/core/middleware/rate_limiting.py +++ b/testbed/core/middleware/rate_limiting.py @@ -1,232 +1,260 @@ """ -Rate limiting middleware for LOLA compliance. +Rate limiting middleware for LOLA source-server compliance. -Per LOLA specification: "The source server MAY rate limit requests by sending -a 429 Too Many Requests response as defined in RFC6585, with a Retry-After header." +LOLA Section 6.7, "Load Management During Fetching of Content": -This middleware implements basic rate limiting for OAuth endpoints to ensure -production-ready behavior for real-world LOLA account portability usage. + The source server MAY rate limit requests by sending a 429 Too Many Requests + response as defined in [RFC6585], with a Retry-After header. If the destination + receives a 429 response status code, it SHOULD respect the Retry-After header + and resume its requests after the chosen delay. + +The normative SHOULD lands on the destination server honoring Retry-After -- which is what +this testbed exists to let destination implementers exercise. That makes predictable, +correlatable 429s more valuable here than accurate ones: a destination cannot develop backoff +logic against a signal that fires for reasons unrelated to its own request pattern. + +See docs/lola-rate-limiting.md for the design rationale. + +What this middleware guarantees: +- A 429 carries Retry-After, a JSON response, CORS headers letting browser-based + federation clients actually read Retry-After. +- Counting is per (rule, client IP). A request only consumes the budget of the + rule matching its own path, never another path's. + +A fixed window was implemented. Each (rule, ip) pair holds an integer counter stored +with a TTL equal to the rule's window. The first request of a window creates it with +cache.add(); later requests use cache.incr(). Expiry is the cache's responsibility, +so there is no cleanup pass and no unbounded growth. + +It relies on incr() preserving the key's TTL, so that continued traffic cannot extend a +window (hammering while blocked must not lengthen a client's own lockout, or Retry-After +becomes a lie). + +Counters live in Django's cache, which under the default LocMemCache is per-process, +so the effective ceiling is: + + limit x gunicorn_workers x running_cloud_run_instances + +Threads do not multiply it -- they share one process's memory, which is exactly why +incr() has to be atomic. Worker processes and Cloud Run instances do. The container +currently runs --workers 1, so today this reduces to limit x instances. """ -import time import logging -from collections import defaultdict -from datetime import datetime, timedelta -from django.http import HttpResponse +import time + from django.conf import settings +from django.core.cache import cache + +from ..utils.errors import build_rate_limit_error logger = logging.getLogger(__name__) -class BasicRateLimitingMiddleware: +CACHE_KEY_PREFIX = "ratelimit" + +# Set once per process by _log_chain_shape_once below. +_chain_shape_logged = False + + +def _log_chain_shape_once(entries, depth, client_ip): """ - Simple in-memory rate limiting middleware for LOLA OAuth endpoints. - - This middleware tracks request rates per IP address and returns RFC6585-compliant - 429 responses with Retry-After headers when rate limits are exceeded. - - Focuses on OAuth authorization endpoints which are most critical for LOLA - account portability operations. + Report the X-Forwarded-For shape this process actually observes, once. + + RATE_LIMIT_TRUSTED_PROXY_DEPTH cannot be validated from code -- only a real + request through the real proxy chain reveals its length, and a wrong value + fails silently when the chain is LONGER than configured. Without this the + resolved IP is visible only when a 429 fires, so checking the setting would mean + deliberately rate limiting production traffic. + + Once per process rather than once globally, so every Cloud Run instance reports + the chain it sees while keeping the log quiet. """ - + global _chain_shape_logged + + if _chain_shape_logged: + return + + _chain_shape_logged = True + logger.info( + "Rate limit client resolution: xff_entries=%s configured_depth=%s resolved=%s", + len(entries), + depth, + client_ip, + ) + + +class RateLimitingMiddleware: + """ + Per-(rule, client IP) fixed-window rate limiting. Applied site-wide: unmatched + paths fall to RATE_LIMIT_DEFAULT, so this governs more than the LOLA surface. + + Settings consumed (testbed/settings/base.py): + RATE_LIMIT_ENABLED (bool) -- optional, defaults True + RATE_LIMIT_RULES (list[dict]) -- optional, defaults [] + RATE_LIMIT_DEFAULT (dict) -- REQUIRED, rule for unmatched paths + RATE_LIMIT_EXEMPT_PREFIXES (list[str]) -- REQUIRED, paths that never count + RATE_LIMIT_TRUSTED_PROXY_DEPTH (int) -- optional, defaults 0 + """ + def __init__(self, get_response): self.get_response = get_response - - # In-memory storage for rate limiting (production would use database) - self.request_counts = defaultdict(list) # IP -> list of request timestamps - - # Rate limiting configuration - # OAuth endpoints get stricter limits since they're more sensitive - self.rate_limits = { - # OAuth authorization endpoint (most critical for LOLA) - '/oauth/authorize/': {'requests': 10, 'window': 300}, # 10 requests per 5 minutes - '/oauth/token/': {'requests': 20, 'window': 300}, # 20 requests per 5 minutes - - # LOLA discovery endpoints - '/.well-known/oauth-authorization-server': {'requests': 30, 'window': 60}, # 30 per minute - - # LOLA collection endpoints (less strict, but still limited) - '/api/actors/': {'requests': 100, 'window': 60}, # 100 requests per minute - } - - # Default rate limit for other endpoints - self.default_limit = {'requests': 200, 'window': 60} # 200 requests per minute - + def __call__(self, request): - # Check if this request should be rate limited + if not self._enabled(): + return self.get_response(request) + + path = request.path + + if self._is_exempt(path): + return self.get_response(request) + + rule = self._rule_for_path(path) client_ip = self.get_client_ip(request) - current_time = time.time() - - # Clean up old entries to prevent memory bloat - self.cleanup_old_entries(current_time) - - # Check rate limit for this request - rate_limit_result = self.check_rate_limit(request, client_ip, current_time) - - if rate_limit_result['exceeded']: - # Return 429 Too Many Requests with Retry-After header - retry_after = rate_limit_result['retry_after'] - + key = f"{CACHE_KEY_PREFIX}:{rule['name']}:{client_ip}" + + try: + allowed, retry_after = self._consume(key, rule["limit"], rule["window"]) + except Exception: + # Fail open. A limiter is a dampening signal under a Section 6.7 MAY; + # a cache failure must never take down the endpoints it protects. + logger.exception( + "Rate limiting check failed, allowing request path=%s rule=%s", + path, + rule["name"], + ) + return self.get_response(request) + + if not allowed: logger.warning( - f"Rate limit exceeded for IP {client_ip} on {request.path}. " - f"Retry after {retry_after} seconds." + "Rate limit exceeded ip=%s rule=%s path=%s limit=%s window=%ss " + "retry_after=%ss", + client_ip, + rule["name"], + path, + rule["limit"], + rule["window"], + retry_after, ) - - response = HttpResponse( - f"Rate limit exceeded. Try again in {retry_after} seconds.", - status=429, - content_type='text/plain' + return build_rate_limit_error( + retry_after_seconds=retry_after, + request=request, + limit=rule["limit"], + window=rule["window"], ) - response['Retry-After'] = str(retry_after) - - # Add CORS headers for ActivityPub federation compatibility - response['Access-Control-Allow-Origin'] = '*' - response['Access-Control-Expose-Headers'] = 'Retry-After' - - return response - - # Record this request and continue - self.request_counts[client_ip].append(current_time) - - response = self.get_response(request) - return response - - def get_client_ip(self, request): - """ - Get client IP address, handling proxy headers. - - In production, this should be configured based on our proxy setup. - """ - # Check for forwarded IP (common with reverse proxies) - x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') - if x_forwarded_for: - return x_forwarded_for.split(',')[0].strip() - - # Check for real IP (some proxy configurations) - x_real_ip = request.META.get('HTTP_X_REAL_IP') - if x_real_ip: - return x_real_ip - - # Fallback to direct connection IP - return request.META.get('REMOTE_ADDR', '127.0.0.1') - - def check_rate_limit(self, request, client_ip, current_time): - """ - Check if the request should be rate limited. - - Returns dict with 'exceeded' boolean and 'retry_after' seconds. - """ - # Find the most specific rate limit for this path - rate_limit = self.get_rate_limit_for_path(request.path) - - # Get request timestamps for this IP - request_times = self.request_counts[client_ip] - - # Filter to requests within the time window - window_start = current_time - rate_limit['window'] - recent_requests = [t for t in request_times if t > window_start] - - # Check if limit is exceeded - if len(recent_requests) >= rate_limit['requests']: - # Calculate when the oldest request in the window will expire - oldest_request = min(recent_requests) - retry_after = int(oldest_request + rate_limit['window'] - current_time) + 1 - - return { - 'exceeded': True, - 'retry_after': max(retry_after, 1) # At least 1 second - } - - return {'exceeded': False, 'retry_after': 0} - - def get_rate_limit_for_path(self, path): + + return self.get_response(request) + + # Configuration accessors + + def _enabled(self): + return bool(getattr(settings, "RATE_LIMIT_ENABLED", True)) + + def _rules(self): + return getattr(settings, "RATE_LIMIT_RULES", []) + + def _fallback_rule(self): + return settings.RATE_LIMIT_DEFAULT + + def _exempt_prefixes(self): + return settings.RATE_LIMIT_EXEMPT_PREFIXES + + # Path resolution + + def _is_exempt(self, path): + return any(path.startswith(prefix) for prefix in self._exempt_prefixes()) + + def _rule_for_path(self, path): """ - Get the rate limit configuration for a specific path. - - Uses the most specific match (longest matching prefix). + Return the rule governing `path`, by longest matching prefix. + + Longest-match means a more specific prefix beats a more general one + regardless of declaration order, so rules can be listed in any sequence. """ - best_match = self.default_limit - best_match_length = 0 - - for pattern, limit in self.rate_limits.items(): - if path.startswith(pattern) and len(pattern) > best_match_length: - best_match = limit - best_match_length = len(pattern) - - return best_match - - def cleanup_old_entries(self, current_time, max_age=3600): + best_rule = self._fallback_rule() + best_length = 0 + + for rule in self._rules(): + prefix = rule["prefix"] + if path.startswith(prefix) and len(prefix) > best_length: + best_rule = rule + best_length = len(prefix) + + return best_rule + + # Client identity + + def get_client_ip(self, request): """ - Remove old entries to prevent memory bloat. - - Removes entries older than max_age seconds (default: 1 hour). + Resolve the client IP by counting in from the RIGHT of X-Forwarded-For. + + Each proxy appends what it observed, so right-hand entries are facts while + the leftmost is whatever the client sent. Reading index 0 would let a client + mint a fresh bucket per request just by varying the header. + + RATE_LIMIT_TRUSTED_PROXY_DEPTH is how many proxies append in front of this + app; the real client sits that many entries in from the right. Default 0 + means the header is not trusted at all, so each deployment opts in. + + Falls back to REMOTE_ADDR when the chain is shorter than the configured + depth -- the header is then not the shape this deployment expects. + + Side effect: reports the observed chain shape once per process (see + _log_chain_shape_once), so the configured depth can be checked without + forcing a 429. """ - cutoff_time = current_time - max_age - - for ip in list(self.request_counts.keys()): - # Filter out old entries - self.request_counts[ip] = [ - t for t in self.request_counts[ip] if t > cutoff_time - ] - - # Remove IP if no recent requests - if not self.request_counts[ip]: - del self.request_counts[ip] - - -class LOLARateLimitingMiddleware(BasicRateLimitingMiddleware): - """ - LOLA-specific rate limiting middleware with enhanced configuration. - - This extends the basic rate limiting with LOLA-specific considerations: - - More restrictive limits for OAuth endpoints - - Special handling for LOLA discovery endpoints - - ActivityPub federation-friendly error responses - """ - - def __init__(self, get_response): - super().__init__(get_response) - - # Override with LOLA-specific rate limits - self.rate_limits.update({ - # Very strict OAuth limits for production LOLA usage - '/oauth/authorize/': {'requests': 5, 'window': 300}, # 5 requests per 5 minutes - '/oauth/token/': {'requests': 10, 'window': 300}, # 10 requests per 5 minutes - - # LOLA discovery endpoints (moderate limits) - '/.well-known/oauth-authorization-server': {'requests': 20, 'window': 60}, - - # LOLA collection endpoints (per LOLA spec considerations) - '/api/actors/': {'requests': 50, 'window': 60}, # 50 requests per minute - }) - - # More conservative default for LOLA production usage - self.default_limit = {'requests': 100, 'window': 60} # 100 requests per minute - - def check_rate_limit(self, request, client_ip, current_time): + depth = int(getattr(settings, "RATE_LIMIT_TRUSTED_PROXY_DEPTH", 0) or 0) + forwarded_for = request.META.get("HTTP_X_FORWARDED_FOR", "") + entries = [part.strip() for part in forwarded_for.split(",") if part.strip()] + + if depth > 0 and len(entries) > depth: + client_ip = entries[len(entries) - depth - 1] + else: + if depth > 0 and entries: + logger.warning( + "X-Forwarded-For shorter than RATE_LIMIT_TRUSTED_PROXY_DEPTH " + "(%s entries, depth %s); falling back to REMOTE_ADDR", + len(entries), + depth, + ) + client_ip = request.META.get("REMOTE_ADDR") or "unknown" + + _log_chain_shape_once(entries, depth, client_ip) + return client_ip + + # Counting + + def _consume(self, key, limit, window): """ - Enhanced rate limit checking with LOLA-specific logic. + Count one request against `key`; return (allowed, retry_after_seconds). + + A companion ":reset" entry records when the window ends, so Retry-After + reports true remaining time rather than a whole window -- destinations SHOULD + honor that value (Section 6.7), so its accuracy is user-visible. + + Blocked requests still increment, which is harmless: incr() preserves the + TTL, so hammering cannot extend a client's own lockout. """ - result = super().check_rate_limit(request, client_ip, current_time) - - # Log rate limiting events for LOLA monitoring - if result['exceeded']: - logger.info( - f"LOLA rate limit triggered: IP={client_ip}, path={request.path}, " - f"retry_after={result['retry_after']}s" - ) - - return result + reset_key = f"{key}:reset" + now = time.time() + # add() succeeds only if the key is absent or expired: this opens a new + # window. Atomic, so concurrent first-requests cannot both win. + if cache.add(key, 1, window): + cache.set(reset_key, now + window, window) + return True, 0 -# Configuration helper for settings.py -def get_rate_limiting_middleware(): - """ - Helper function to get the appropriate rate limiting middleware class. - - Can be used in settings.py to choose between basic and LOLA-specific middleware. - """ - # Use LOLA-specific middleware if explicitly configured - if getattr(settings, 'USE_LOLA_RATE_LIMITING', False): - return 'testbed.core.middleware.rate_limiting.LOLARateLimitingMiddleware' - else: - return 'testbed.core.middleware.rate_limiting.BasicRateLimitingMiddleware' + try: + count = cache.incr(key) + except ValueError: + # Expired between add() and incr(); treat as a fresh window. + cache.set(key, 1, window) + cache.set(reset_key, now + window, window) + return True, 0 + + if count > limit: + reset_at = cache.get(reset_key) + # Reset marker gone: a full window is the safe over-estimate. + retry_after = max(int(reset_at - now) + 1, 1) if reset_at else window + return False, retry_after + + return True, 0 diff --git a/testbed/core/tests/test_rate_limiting.py b/testbed/core/tests/test_rate_limiting.py new file mode 100644 index 0000000..00d378f --- /dev/null +++ b/testbed/core/tests/test_rate_limiting.py @@ -0,0 +1,439 @@ +import json +import logging + +import pytest +from django.core.cache import cache +from django.http import HttpResponse +from django.test import RequestFactory, override_settings + +from testbed.core.middleware import rate_limiting +from testbed.core.middleware.rate_limiting import RateLimitingMiddleware + +LOGGER_NAME = "testbed.core.middleware.rate_limiting" + +TEST_RULES = [ + {"name": "oauth_authorize", "prefix": "/oauth/authorize/", "limit": 2, "window": 300}, + {"name": "lola_api", "prefix": "/api/actors/", "limit": 5, "window": 60}, +] + +TEST_DEFAULT = {"name": "default", "limit": 50, "window": 60} + +rate_limiting_enabled = override_settings( + RATE_LIMIT_ENABLED=True, + RATE_LIMIT_RULES=TEST_RULES, + RATE_LIMIT_DEFAULT=TEST_DEFAULT, + RATE_LIMIT_EXEMPT_PREFIXES=["/static/", "/health"], + RATE_LIMIT_TRUSTED_PROXY_DEPTH=0, +) + +CLIENT_IP = "203.0.113.5" +OTHER_IP = "198.51.100.7" + +AUTHORIZE = "/oauth/authorize/" + + +@pytest.fixture(autouse=True) +def clear_rate_limit_cache(): + cache.clear() + yield + cache.clear() + + +def make_middleware(): + """ + Return (middleware, downstream_paths). + + downstream_paths records every request that reached the wrapped view, so tests + can distinguish "allowed" from "short-circuited with 429" without relying only + on status codes. + """ + downstream_paths = [] + + def get_response(request): + downstream_paths.append(request.path) + return HttpResponse("ok") + + return RateLimitingMiddleware(get_response), downstream_paths + + +def send(middleware, path, ip=CLIENT_IP, **extra): + # Drive a single GET through the middleware and return the response + request = RequestFactory().get(path, REMOTE_ADDR=ip, **extra) + return middleware(request) + + +def exhaust(middleware, path=AUTHORIZE, ip=CLIENT_IP, times=2): + # Consume a rule's whole budget so the next request is rejected + for _ in range(times): + send(middleware, path, ip=ip) + + +# Bucket isolation + + +@rate_limiting_enabled +def test_traffic_to_other_paths_does_not_consume_the_oauth_budget(): + """ + Counters are keyed by rule as well as by client. Drop the rule from that key and + every path shares one budget, so unrelated traffic silently eats a stricter + rule's allowance. + + The count is what makes this trigger: 12 requests on the default rule is + well past oauth_authorize's entire budget of 2, so a shared counter would already + be over the line by the time the first OAuth request arrives. Keep it above + oauth_authorize's limit or this stops testing anything. + """ + middleware, downstream = make_middleware() + + # A page load's worth of unrelated requests, all on the default rule + for _ in range(12): + assert send(middleware, "/").status_code == 200 + + # First ever request to the OAuth endpoint must be allowed + assert send(middleware, AUTHORIZE).status_code == 200 + assert downstream[-1] == AUTHORIZE + + +@rate_limiting_enabled +def test_each_rule_keeps_an_independent_budget(): + # Exhausting one rule must leave every other rule untouched + middleware, _ = make_middleware() + + exhaust(middleware) + assert send(middleware, AUTHORIZE).status_code == 429 + + # Other rules are unaffected + assert send(middleware, "/api/actors/1/").status_code == 200 + assert send(middleware, "/").status_code == 200 + + +@rate_limiting_enabled +def test_budgets_are_per_client(): + # One client exhausting a rule must not affect a different client + middleware, _ = make_middleware() + + exhaust(middleware) + assert send(middleware, AUTHORIZE).status_code == 429 + + assert send(middleware, AUTHORIZE, ip=OTHER_IP).status_code == 200 + + +# Limit behaviour + + +@rate_limiting_enabled +def test_limit_boundary_allows_exactly_the_configured_count(): + # A limit of N allows N requests and rejects the N+1th + middleware, downstream = make_middleware() + + assert send(middleware, AUTHORIZE).status_code == 200 + assert send(middleware, AUTHORIZE).status_code == 200 + assert send(middleware, AUTHORIZE).status_code == 429 + + # The rejected request never reached the view + assert downstream.count(AUTHORIZE) == 2 + + +@rate_limiting_enabled +def test_window_expiry_opens_a_fresh_budget(): + """ + Once the TTL lapses the counter is gone and cache.add() starts a new window. + Expiry is the cache's job, so nothing in the middleware resets a counter -- + if the TTL handling broke, blocked clients would simply stay blocked forever. + """ + middleware, _ = make_middleware() + exhaust(middleware) + assert send(middleware, AUTHORIZE).status_code == 429 + + # Force the boundary the TTL would have reached + for key in list(cache._expire_info): + cache._expire_info[key] = 0 + + assert send(middleware, AUTHORIZE).status_code == 200 + assert send(middleware, AUTHORIZE).status_code == 200 + assert send(middleware, AUTHORIZE).status_code == 429 + + +@rate_limiting_enabled +def test_blocked_requests_do_not_extend_the_window(): + """ + incr() must preserve the key's TTL. If it reset it, continued hammering would + keep pushing the window out and Retry-After would become a lie. + """ + middleware, _ = make_middleware() + exhaust(middleware) + + first_retry_after = int(send(middleware, AUTHORIZE)["Retry-After"]) + + # Hammer while blocked + for _ in range(20): + response = send(middleware, AUTHORIZE) + assert response.status_code == 429 + + # Time only moves forward, so Retry-After must not have grown + assert int(response["Retry-After"]) <= first_retry_after + + +# The 429 contract (LOLA Section 6.7 / RFC6585) + + +@rate_limiting_enabled +def test_429_carries_the_standard_json_error_contract(): + # The 429 must be machine-readable, on the same contract as every other error + middleware, _ = make_middleware() + exhaust(middleware) + + response = send(middleware, AUTHORIZE) + + assert response.status_code == 429 + assert response["Content-Type"] == "application/json" + + body = json.loads(response.content) + assert body["error_code"] == "rate_limit_exceeded" + assert body["detail"] + assert body["hint"] + assert body["remediation"] + assert body["endpoint"] == AUTHORIZE + assert body["method"] == "GET" + assert body["request_id"] + assert body["timestamp"] + + +@rate_limiting_enabled +def test_429_carries_retry_after_and_federation_headers(): + """ + Retry-After is what Section 6.7 asks destinations to honor, and browser-based + federation clients cannot read it unless it is explicitly exposed via CORS. + """ + middleware, _ = make_middleware() + exhaust(middleware) + + response = send(middleware, AUTHORIZE) + + assert 1 <= int(response["Retry-After"]) <= 300 # within the configured window + assert response["Cache-Control"] == "no-store" + assert response["Access-Control-Allow-Origin"] == "*" + assert response["Access-Control-Expose-Headers"] == "Retry-After" + + +@rate_limiting_enabled +def test_429_hint_reports_the_limit_that_was_hit(): + # The body should say which budget was exhausted, not just that one was + middleware, _ = make_middleware() + exhaust(middleware) + + hint = json.loads(send(middleware, AUTHORIZE).content)["hint"] + + assert "2" in hint and "300" in hint + + +# Client identification + + +@rate_limiting_enabled +def test_forwarded_for_is_ignored_when_no_proxy_is_trusted(): + """ + At depth 0 the header is not trusted at all, so varying it cannot mint a fresh + bucket. Reading the leftmost entry instead would hand a caller exactly that. + """ + middleware, _ = make_middleware() + + for i in range(2): + response = send(middleware, AUTHORIZE, HTTP_X_FORWARDED_FOR=f"198.51.100.{i}") + assert response.status_code == 200 + + # Same real client, brand new spoofed header: still blocked + response = send(middleware, AUTHORIZE, HTTP_X_FORWARDED_FOR="198.51.100.99") + assert response.status_code == 429 + + +@override_settings(RATE_LIMIT_TRUSTED_PROXY_DEPTH=1) +def test_client_ip_is_counted_in_from_the_right(): + """ + With one trusted trailing entry, the caller is the one just before it. Anything + the caller injects lands further left and must be ignored. + """ + middleware, _ = make_middleware() + factory = RequestFactory() + + # Well-formed chain + request = factory.get( + "/", REMOTE_ADDR="10.0.0.1", HTTP_X_FORWARDED_FOR="203.0.113.5, 130.211.0.1" + ) + assert middleware.get_client_ip(request) == "203.0.113.5" + + # Client injects an entry; the real client is still one in from the right + request = factory.get( + "/", + REMOTE_ADDR="10.0.0.1", + HTTP_X_FORWARDED_FOR="1.1.1.1, 203.0.113.5, 130.211.0.1", + ) + assert middleware.get_client_ip(request) == "203.0.113.5" + + +@override_settings(RATE_LIMIT_TRUSTED_PROXY_DEPTH=1) +def test_short_forwarded_for_chain_falls_back_to_remote_addr(): + """ + A chain shorter than the configured depth means the header is not the shape this + deployment expects, so fall back to the value a client cannot forge. + """ + middleware, _ = make_middleware() + + request = RequestFactory().get( + "/", REMOTE_ADDR="10.0.0.1", HTTP_X_FORWARDED_FOR="1.1.1.1" + ) + assert middleware.get_client_ip(request) == "10.0.0.1" + + +@rate_limiting_enabled +def test_chain_shape_is_reported_once_per_process(caplog, monkeypatch): + """ + RATE_LIMIT_TRUSTED_PROXY_DEPTH can only be validated against a real proxy chain, + and a value that is too low fails silently. This line is the only way to observe + what was resolved without deliberately triggering a 429. + """ + # monkeypatch, not a bare assignment: the flag is module state and would + # otherwise stay set for every test that runs after this one. + monkeypatch.setattr(rate_limiting, "_chain_shape_logged", False) + middleware, _ = make_middleware() + + # REMOTE_ADDR deliberately differs from every X-Forwarded-For entry, so the + # report shows which source was actually used rather than a coincidental match. + forwarded = "203.0.113.5, 10.0.0.1" + with caplog.at_level(logging.INFO, logger=LOGGER_NAME): + send(middleware, AUTHORIZE, ip="192.0.2.77", HTTP_X_FORWARDED_FOR=forwarded) + send(middleware, AUTHORIZE, ip="192.0.2.77", HTTP_X_FORWARDED_FOR=forwarded) + + reports = [ + r.getMessage() for r in caplog.records if "client resolution" in r.getMessage() + ] + + assert len(reports) == 1, "must report once per process, not per request" + # The chain length is reported even at depth 0 -- that is what reveals a + # deployment ignoring entries it ought to be trusting. + assert "xff_entries=2" in reports[0] + assert "configured_depth=0" in reports[0] + assert "resolved=192.0.2.77" in reports[0] # REMOTE_ADDR, not the header + + +# Exemptions + + +@rate_limiting_enabled +def test_exempt_paths_never_consume_a_budget(): + """ + Assets and platform health checks must not spend a user's allowance. In + development DEBUG=True serves static through Django. + """ + middleware, _ = make_middleware() + + for _ in range(50): + assert send(middleware, "/static/css/main.css").status_code == 200 + + for _ in range(50): + assert send(middleware, "/health").status_code == 200 + + # Every other budget is untouched. + assert send(middleware, AUTHORIZE).status_code == 200 + + +# Rule resolution + + +@rate_limiting_enabled +def test_longest_matching_prefix_wins(): + """ + Specificity, not declaration order, decides which rule applies. + + The rules must OVERLAP for this to test anything: with disjoint prefixes no path + matches two rules, and first-match and longest-match become indistinguishable. + Both declaration orders are checked, since order-independence is the point -- + reordering RATE_LIMIT_RULES must never change behaviour. + """ + overlapping = [ + {"name": "lola_api", "prefix": "/api/actors/", "limit": 5, "window": 60}, + { + "name": "migration", + "prefix": "/api/actors/5/migration/", + "limit": 1, + "window": 60, + }, + ] + middleware, _ = make_middleware() + + for rules in (overlapping, list(reversed(overlapping))): + with override_settings(RATE_LIMIT_RULES=rules): + # Only the general prefix matches + assert ( + middleware._rule_for_path("/api/actors/5/outbox/")["name"] == "lola_api" + ) + # Both match; the more specific one must win + assert ( + middleware._rule_for_path("/api/actors/5/migration/content/")["name"] + == "migration" + ) + # Neither matches. + assert middleware._rule_for_path("/somewhere/else")["name"] == "default" + + +# Operational safety + + +@override_settings(RATE_LIMIT_ENABLED=False, RATE_LIMIT_RULES=TEST_RULES) +def test_disabled_middleware_allows_everything(): + middleware, downstream = make_middleware() + + for _ in range(20): + assert send(middleware, AUTHORIZE).status_code == 200 + + assert len(downstream) == 20 + + +@rate_limiting_enabled +def test_cache_failure_fails_open(monkeypatch): + """ + A limiter is a dampening signal under a Section 6.7 MAY. If the cache breaks it + must let traffic through rather than take down the endpoints it protects. + """ + middleware, downstream = make_middleware() + + def explode(*args, **kwargs): + raise RuntimeError("cache unavailable") + + monkeypatch.setattr("testbed.core.middleware.rate_limiting.cache.add", explode) + + assert send(middleware, AUTHORIZE).status_code == 200 + assert downstream == [AUTHORIZE] + + +# Wiring + + +@pytest.mark.django_db +def test_middleware_is_wired_into_the_request_stack(client): + """ + Everything above drives the middleware directly. This proves it is actually + installed in settings.MIDDLEWARE and reached by real requests. + """ + discovery = "/.well-known/oauth-authorization-server" + + with override_settings( + RATE_LIMIT_ENABLED=True, + RATE_LIMIT_RULES=[ + { + "name": "lola_discovery", + "prefix": discovery, + "limit": 1, + "window": 60, + } + ], + RATE_LIMIT_DEFAULT=TEST_DEFAULT, + RATE_LIMIT_EXEMPT_PREFIXES=[], + RATE_LIMIT_TRUSTED_PROXY_DEPTH=0, + ): + assert client.get(discovery).status_code == 200 + + response = client.get(discovery) + assert response.status_code == 429 + assert response["Retry-After"] + assert json.loads(response.content)["error_code"] == "rate_limit_exceeded" diff --git a/testbed/core/utils/errors.py b/testbed/core/utils/errors.py index 658ac38..e45ec1b 100644 --- a/testbed/core/utils/errors.py +++ b/testbed/core/utils/errors.py @@ -1,5 +1,6 @@ import uuid from datetime import timezone, datetime +from django.http import JsonResponse from rest_framework.response import Response """ @@ -21,7 +22,6 @@ class ErrorCodes: FORBIDDEN_ACCESS = "forbidden_access" UNAUTHORIZED = "unauthorized" ACTOR_MISMATCH = "actor_mismatch" - # Rate Limiting Errors (429) RATE_LIMIT_EXCEEDED = "rate_limit_exceeded" @@ -49,13 +49,62 @@ def generate_request_id(): return str(uuid.uuid4()) +def build_error_payload(error_code, detail, request=None, hint=None, remediation=None): + """ + Build the error body shared by every error response. + + Two wrappers call this with the same body: + - build_error_response -> DRF Response for the view layer + - build_rate_limit_error -> JsonResponse for middleware + + They can't share a wrapper because they run in different layers: + - Views need a DRF Response so activitypub_content can use + request.accepted_renderer to set application/activity+json. + - Middleware runs outside DRF, where a Response has no renderer + and raises on serialization, so it needs a plain JsonResponse. + + Args: + error_code (str): Machine-readable error identifier from ErrorCodes + detail (str): Human-readable error description + request (HttpRequest, optional): Django request object for context + hint (str, optional): Additional context or explanation + remediation (str, optional): Actionable steps to fix the error + + Returns: + dict: The error body, ready to be wrapped by either response class + """ + error_data = { + "error_code": error_code, + "detail": detail, + "timestamp": datetime.now(timezone.utc).isoformat(), + } + + # Add optional context fields if provided + if hint: + error_data["hint"] = hint + + if remediation: + error_data["remediation"] = remediation + + if request: + error_data["endpoint"] = request.path + error_data["method"] = request.method + # Generate request ID for this specific request + error_data["request_id"] = generate_request_id() + + return error_data + + def build_error_response(error_code, detail, status_code, request=None, hint=None, remediation=None): """ - Build standardized JSON error response. + Build standardized JSON error response for the DRF view layer. Creates consistent, developer-friendly error responses with comprehensive metadata for debugging, remediation, and support purposes. + The body is built by build_error_payload; this function only wraps it in the + response class the view layer needs. + Args: error_code (str): Machine-readable error identifier from ErrorCodes detail (str): Human-readable error description @@ -77,26 +126,16 @@ def build_error_response(error_code, detail, status_code, request=None, hint=Non ... remediation="Request OAuth token with 'activitypub_account_portability' scope" ... ) """ - error_data = { - "error_code": error_code, - "detail": detail, - "timestamp": datetime.now(timezone.utc).isoformat(), - } - - # Add optional context fields if provided - if hint: - error_data["hint"] = hint - - if remediation: - error_data["remediation"] = remediation - - if request: - error_data["endpoint"] = request.path - error_data["method"] = request.method - # Generate request ID for this specific request - error_data["request_id"] = generate_request_id() - - return Response(error_data, status=status_code) + return Response( + build_error_payload( + error_code=error_code, + detail=detail, + request=request, + hint=hint, + remediation=remediation, + ), + status=status_code, + ) def build_actor_not_found_error(actor_id, request=None): @@ -163,27 +202,53 @@ def build_actor_mismatch_error(request=None): ) -def build_rate_limit_error(retry_after_seconds, request=None): +def build_rate_limit_error(retry_after_seconds, request=None, limit=None, window=None): """ - Build standardized 429 error for rate limiting with Retry-After header. + Build the complete 429 response for rate-limited requests (LOLA Section 6.7). + + Returns a JsonResponse rather than a DRF Response, unlike every other builder in + this module, because rate limiting is enforced in middleware -- outside the DRF + view layer, where a Response has no renderer and raises on serialization. + + - Retry-After: LOLA Section 6.7 describes rate limiting as a 429 "with a + Retry-After header", and destinations SHOULD honor it before resuming. Args: - retry_after_seconds (int): Seconds to wait before retrying + retry_after_seconds (int): Seconds until the current window resets request (HttpRequest, optional): Django request object for context + limit (int, optional): Configured request ceiling + window (int, optional): Window length in seconds Returns: - Response: 429 error response with rate limit context + JsonResponse: 429 response with Retry-After and federation-safe CORS headers """ - response = build_error_response( + if limit is not None and window is not None: + hint = ( + f"Too many requests: the limit is {limit} per {window} seconds. " + f"Please wait {retry_after_seconds} seconds before retrying" + ) + else: + hint = f"Too many requests. Please wait {retry_after_seconds} seconds before retrying" + + payload = build_error_payload( error_code=ErrorCodes.RATE_LIMIT_EXCEEDED, detail="Request rate limit exceeded", - status_code=429, request=request, - hint=f"Too many requests. Please wait {retry_after_seconds} seconds before retrying", - remediation="Implement exponential backoff or reduce request frequency" + hint=hint, + remediation="Honor the Retry-After header, then resume with exponential backoff", ) - # Add standard Retry-After header for rate limiting - response['Retry-After'] = str(retry_after_seconds) + response = JsonResponse(payload, status=429) + + # Standard rate-limiting header (RFC6585 / LOLA Section 6.7) + response["Retry-After"] = str(retry_after_seconds) + + # Never let a 429 be replayed from a cache after the client has backed off + response["Cache-Control"] = "no-store" + + # CORS for ActivityPub federation: Retry-After is unreadable to browser + # clients unless it is explicitly exposed + response["Access-Control-Allow-Origin"] = "*" + response["Access-Control-Expose-Headers"] = "Retry-After" return response diff --git a/testbed/settings/base.py b/testbed/settings/base.py index 02922c6..084b27d 100644 --- a/testbed/settings/base.py +++ b/testbed/settings/base.py @@ -62,7 +62,7 @@ "django.contrib.sessions.middleware.SessionMiddleware", "django.middleware.common.CommonMiddleware", # LOLA Rate Limiting - positioned early to protect all endpoints - "testbed.core.middleware.rate_limiting.BasicRateLimitingMiddleware", + "testbed.core.middleware.rate_limiting.RateLimitingMiddleware", "django.middleware.csrf.CsrfViewMiddleware", "django.contrib.auth.middleware.AuthenticationMiddleware", "django.contrib.messages.middleware.MessageMiddleware", @@ -259,3 +259,57 @@ 'rest_framework.authentication.SessionAuthentication', ], } + +# Rate limiting (LOLA Section 6.7) + +# Counters are per (rule, client IP) and live in the cache, so they are per-process. +# See testbed/core/middleware/rate_limiting.py and docs/lola-rate-limiting.md for the design rationale. + +RATE_LIMIT_ENABLED = True + +# Longest matching prefix wins, so declaration order does not matter. +# Limits are intentionally generous: one interactive OAuth authorization +# spans several requests, and this testbed exists for people to exercise that flow repeatedly. +RATE_LIMIT_RULES = [ + { + "name": "oauth_authorize", + "prefix": "/oauth/authorize/", + "limit": 60, + "window": 300, + }, + { + "name": "oauth_token", + "prefix": "/oauth/token/", + "limit": 120, + "window": 300, + }, + { + "name": "lola_discovery", + "prefix": "/.well-known/oauth-authorization-server", + "limit": 60, + "window": 60, + }, + { + "name": "lola_api", + "prefix": "/api/actors/", + "limit": 120, + "window": 60, + }, +] + +# Applied to any path not matched by a rule above. +RATE_LIMIT_DEFAULT = {"name": "default", "limit": 300, "window": 60} + +# Paths that never consume a budget +RATE_LIMIT_EXEMPT_PREFIXES = [ + "/static/", + "/media/", + "/health", + "/favicon.ico", +] + +# Number of proxies that append to X-Forwarded-For in front. +# 0 means the header is not trusted at all and REMOTE_ADDR is used instead; each +# deployment opts in by declaring how deep its own chain is. Set for Cloud Run in +# production.py / staging.py. +RATE_LIMIT_TRUSTED_PROXY_DEPTH = 0 diff --git a/testbed/settings/development.py b/testbed/settings/development.py index 06fbe33..d1d3980 100644 --- a/testbed/settings/development.py +++ b/testbed/settings/development.py @@ -14,6 +14,8 @@ ALLOWED_HOSTS = ["localhost", "127.0.0.1"] BASE_URL = "http://localhost:8000" +RATE_LIMIT_ENABLED = env.bool("DJANGO_RATE_LIMIT_ENABLED", default=False) + # Seeding settings SEED_ADMIN_USERNAME = "admin" SEED_ADMIN_EMAIL = "admin@seeding.com" diff --git a/testbed/settings/production.py b/testbed/settings/production.py index 7b6a26a..9be3c52 100644 --- a/testbed/settings/production.py +++ b/testbed/settings/production.py @@ -13,6 +13,40 @@ # Cloud Run uses X-Forwarded-Proto header for HTTPS detection SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') +""" +Who counts as "the client" for rate limiting. + +Direct public Cloud Run traffic always passes through Google's managed frontend, +so REMOTE_ADDR is that frontend rather than the caller and must not be assumed to be +the original caller. The caller's real address is in X-Forwarded-For instead. That header is a +list, and each machine appends what it saw -- so the entries on the RIGHT are +trustworthy and the ones on the LEFT are whatever the caller typed. + +Cloud Run does not document a stable, trusted X-Forwarded-For layout for this +run.app/domain-mapping path. (The ", " format is specified for +external Application Load Balancers, which this testbed does not use yet.) + +This number says how many entries on the right belong to Google. The caller is the +one just before them. + + 1 -> ", " picks + 0 -> ignore the header, use REMOTE_ADDR (= Google, so everyone shares + a single rate-limit bucket) + +Why 1 when Google does not document the exact layout for this run.app / +domain-mapping path: for honest callers 1 is never worse than 0. If the layout is +what we expect, 1 identifies each caller correctly. If it is not, 1 falls back to +REMOTE_ADDR and behaves exactly like 0. + +The risk of 1 is that a caller could pad the header to get a fresh bucket every +request and dodge the limit -- and nothing logs when that happens. + +To check it after deploy: make one request with NO X-Forwarded-For, then look for +"client resolution: xff_entries=N" in the logs. Set this to N - 1. +See docs/lola-rate-limiting.md. +""" +RATE_LIMIT_TRUSTED_PROXY_DEPTH = env.int("DJANGO_RATE_LIMIT_TRUSTED_PROXY_DEPTH", default=1) + # PostgreSQL for production DATABASES = {"default": env.db_url("DJ_DATABASE_CONN_STRING")} diff --git a/testbed/settings/test.py b/testbed/settings/test.py index fe97d91..a609242 100644 --- a/testbed/settings/test.py +++ b/testbed/settings/test.py @@ -20,6 +20,9 @@ } } +# test_rate_limiting.py re-enables it explicitly with override_settings +RATE_LIMIT_ENABLED = False + # Faster password hashing for tests PASSWORD_HASHERS = [ "django.contrib.auth.hashers.MD5PasswordHasher",