Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .deployment-state.json

This file was deleted.

Binary file modified data/webhook-dlq.db
Binary file not shown.
88 changes: 83 additions & 5 deletions docs/request-limits-implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,67 @@ ALLOWED_CONTENT_TYPES=application/json
REQUEST_LIMITS_EXCLUDE_PATHS=/health,/metrics
```

### Rate Limiting (RFC 6585)

Rate limiting is enforced by the `src/middleware/rateLimiter.ts` middleware, which returns RFC 6585 compliant 429 Too Many Requests responses.

#### 429 Response Format

All 429 responses include:
- **HTTP Status**: 429 Too Many Requests
- **Retry-After Header**: Seconds to wait before retrying (required by RFC 6585)
- **X-RateLimit-* Headers**: Current rate limit state
- `X-RateLimit-Limit`: Maximum requests allowed in the window
- `X-RateLimit-Remaining`: Requests remaining in current window
- `X-RateLimit-Reset`: Seconds until window resets
- `X-RateLimit-Blocked`: `true` if client is hard-blocked (abuse detected)

#### Response Body (Safe Error Contract)

```json
{
"error": {
"code": "rate_limited",
"message": "Too many requests — please try again later",
"requestId": "unique-correlation-id"
}
}
```

**Security Notes:**
- Error messages follow the safe-error policy (`src/errors/safeErrors.ts`)
- No internal state or implementation details are leaked
- Messages are consistent regardless of block reason (rate limit vs. abuse)
- `requestId` allows clients to correlate with server logs

#### Client Backoff Guidance

Clients encountering 429 responses should:
1. **Check the `Retry-After` header** for the recommended wait time (in seconds)
2. **Respect exponential backoff**: Each successive violation doubles the block duration
3. **Monitor `X-RateLimit-Remaining`** on successful requests to pace load
4. **Read `X-RateLimit-Reset`** to understand when the window resets

Example client retry logic:
```javascript
async function makeRequestWithBackoff(url, options = {}) {
for (let attempt = 1; attempt <= 3; attempt++) {
const response = await fetch(url, options);

if (response.status === 429) {
const retryAfter = response.headers.get('Retry-After');
const waitMs = (parseInt(retryAfter, 10) || 60) * 1000;
console.log(`Rate limited. Waiting ${waitMs}ms before retry...`);
await new Promise(r => setTimeout(r, waitMs));
continue;
}

return response;
}
throw new Error('Rate limited after 3 attempts');
}
```

### Error Responses

All errors use the standard application error format:
Expand All @@ -71,6 +132,8 @@ All errors use the standard application error format:
- Integration tests: Full application testing
- Environment configuration tests
- Error handling validation
- Rate limit header verification
- Safe-error contract assertion

### Running Tests
```bash
Expand All @@ -81,27 +144,31 @@ npm run test:watch
### Specific Test Files
- `src/middleware/__tests__/requestLimits.test.ts` - Unit tests
- `src/requestLimits.integration.test.ts` - Integration tests
- `src/rateLimit.integration.test.ts` - Rate limiting integration tests

## Security Benefits

1. **DoS Prevention**: Limits request size to prevent memory exhaustion
2. **Content-Type Security**: Prevents parsing vulnerabilities
3. **Standardized Errors**: Consistent error handling prevents information leakage
4. **Configurable**: Environment-driven settings for different deployment needs
3. **Rate Limiting**: Protects against abuse and brute-force attacks
4. **Standardized Errors**: Consistent error handling prevents information leakage (RFC 6585)
5. **Configurable**: Environment-driven settings for different deployment needs

## Performance Impact

- Minimal overhead (header-only validation when possible)
- Early rejection of invalid requests
- No request body buffering
- Efficient string comparisons
- In-process rate limit store (no Redis dependency required)

## Migration Guide

### For Existing Applications
1. No breaking changes for valid requests
2. Invalid requests will now be rejected with proper error codes
3. Configure environment variables as needed
3. Rate-limited requests return RFC 6585 compliant 429 responses
4. Configure environment variables as needed

### Recommended Settings
- **Development**: Default settings (1MB, JSON-only)
Expand All @@ -113,13 +180,16 @@ npm run test:watch
### Key Metrics to Monitor
- Rate of 413 errors (payload too large)
- Rate of 415 errors (unsupported media type)
- Rate of 429 errors (rate limited)
- Average request size
- Request size distribution
- Number of hard-blocked clients

### Alerting
- High error rates may indicate attacks
- Monitor for abuse patterns
- Track geographic distribution of violations
- Alert on sustained 429 response rates

## Troubleshooting

Expand All @@ -130,19 +200,27 @@ npm run test:watch
- Verify content-type configuration
- Consider path exclusions if needed

2. **Integration issues**
2. **Clients receiving 429 responses**
- Verify client is respecting `Retry-After` header
- Check `X-RateLimit-Remaining` to understand budget
- Review abuse patterns in logs

3. **Integration issues**
- Ensure clients send proper Content-Type headers
- Verify request sizes are within limits
- Check environment variable configuration
- Verify `requestId` correlation in logs

### Debugging
- Enable debug logging
- Monitor error responses
- Monitor error responses and headers
- Test with different request sizes and content-types
- Simulate rate limit scenarios with load testing

## Future Considerations

1. **Dynamic Limits**: Per-endpoint configuration
2. **Rate Limiting Integration**: Combined validation
3. **Advanced Content-Type**: Schema validation
4. **Machine Learning**: Adaptive limit adjustment
5. **Distributed Rate Limiting**: Redis-backed store for multi-replica deployments
42 changes: 32 additions & 10 deletions src/middleware/rateLimiter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,18 @@
* ## Algorithm
* Uses a **sliding-window counter** per key (default: client IP).
* When a key exceeds `maxRequests` within `windowMs`:
* 1. A 429 response is returned immediately.
* 1. A 429 Too Many Requests response is returned immediately.
* 2. The abuse guard checks whether the violation count itself exceeds
* `abuseThreshold`. If so, the key is **hard-blocked** for `blockDurationMs`.
*
* ## RFC 6585 Compliance
* All 429 responses include:
* - **HTTP Status**: 429 Too Many Requests
* - **Retry-After Header**: Seconds until the rate limit resets or block expires
* - **X-RateLimit-* Headers**: Current state (Limit, Remaining, Reset)
* - **Safe Error Body**: Conforms to the error contract in src/errors/safeErrors.ts
* to prevent information disclosure via error messages.
*
* ## Adaptive throttling
* The abuse guard doubles the block duration on every successive violation
* (exponential back-off), up to `maxBlockDurationMs`.
Expand All @@ -23,6 +31,7 @@
* - Keys are hashed in the store — raw IPs are never persisted.
* - All timing operations use `Date.now()` (monotonic in V8 ≥ Node 16).
* - Blocked responses include `Retry-After` to aid legitimate clients.
* - Error messages are sanitized per the safe-error policy (CWE-209).
* - Headers expose only aggregate counts, never raw keys.
*
* @example
Expand All @@ -36,6 +45,7 @@

import { Request, Response, NextFunction } from 'express';
import { RateLimitStore } from '../lib/rateLimitStore';
import { sanitizeErrorMessage } from '../errors/safeErrors';


export interface RateLimiterConfig {
Expand Down Expand Up @@ -147,12 +157,16 @@ export function createRateLimiter(config: RateLimiterConfig = {}) {
res.setHeader('X-RateLimit-Blocked', 'true');
}
const requestId = typeof res.locals.requestId === 'string' ? res.locals.requestId : 'unknown';
const code = 'rate_limited';
const safeMessage = sanitizeErrorMessage(
'Too many requests — please try again later',
code,
);
res.status(429).json({
error: {
code: 'rate_limited',
message: 'Your access has been temporarily blocked due to excessive requests.',
code,
message: safeMessage,
requestId,
retryAfter: retryAfterSec,
},
});
return;
Expand Down Expand Up @@ -222,12 +236,16 @@ export function createRateLimiter(config: RateLimiterConfig = {}) {
res.setHeader('X-RateLimit-Blocked', 'true');
}
const requestId = typeof res.locals.requestId === 'string' ? res.locals.requestId : 'unknown';
const code = 'rate_limited';
const safeMessage = sanitizeErrorMessage(
'Too many requests — please try again later',
code,
);
res.status(429).json({
error: {
code: 'rate_limited',
message: 'Abuse detected. Your access has been temporarily blocked.',
code,
message: safeMessage,
requestId,
retryAfter: retryAfterSec,
},
});
return;
Expand All @@ -237,12 +255,16 @@ export function createRateLimiter(config: RateLimiterConfig = {}) {

if (sendHeaders) res.setHeader('Retry-After', resetSec);
const requestId = typeof res.locals.requestId === 'string' ? res.locals.requestId : 'unknown';
const code = 'rate_limited';
const safeMessage = sanitizeErrorMessage(
'Too many requests — please try again later',
code,
);
res.status(429).json({
error: {
code: 'rate_limited',
message: `Rate limit exceeded. Try again in ${resetSec} second(s).`,
code,
message: safeMessage,
requestId,
retryAfter: resetSec,
},
});
return;
Expand Down
Binary file modified talenttrust.db
Binary file not shown.
Loading