[surrealwolf]

Overview

Implements exponential backoff retry logic to handle HTTP 429 (Too Many Requests) errors from the Hell Divers 2 API.

Problem

The data collector was hitting rate limit errors (429) on dispatches and other endpoints during collection cycles. The original 2-second delay between requests wasn't sufficient for the 5 requests/10 seconds API limit.

Solution

• Added _fetch_with_backoff() method with exponential backoff on 429 errors
• Backoff schedule: 2s → 4s → 8s → 16s → 32s (up to 5 retries)
• Updated all 10 scraper methods to use automatic retry logic
• Gracefully handles rate limit errors with transparent logging
• Maintains type safety with isinstance() checks for dict/list returns

Changes

• src/scraper.py: Refactored all HTTP requests to use centralized backoff handler
• Simplified method implementations (removed duplicate try/except blocks)
• Added warning logs with retry attempt information

Testing

• ✅ 11/11 tests passing
• ✅ Ruff linting: All checks passed
• ✅ Mypy type checking: 0 issues found

Related Issues

Fixes issue where collection cycle fails with "429 Client Error: Too Many Requests"

[Copilot]

Pull Request Overview

This PR implements exponential backoff retry logic to handle HTTP 429 (Too Many Requests) errors from the Hell Divers 2 API, addressing rate limit violations during data collection cycles.

Key Changes:

• Introduced centralized _fetch_with_backoff() method with exponential backoff (2s → 4s → 8s → 16s → 32s over 5 retries)
• Refactored all 10 scraper methods to use the new backoff handler
• Added type safety checks with isinstance() for list-returning endpoints