diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ccc4f88..799fd98 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,8 +1,38 @@ # High Command API - AI Coding Agent Instructions ## Project Overview -**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, statistics, and faction data via background tasks and serves them through a comprehensive REST API. - +**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, major orders, dispatches, planet events, statistics, and faction data via background tasks and serves them through a comprehensive REST API. + +**API Source**: Community-maintained Hell Divers 2 API at `https://api.helldivers2.dev/api/v1` (popular, actively maintained) +- Rate limit: 5 requests per 10 seconds +- Required headers: X-Super-Client, X-Super-Contact +- Status: Live and actively maintained with real-time data + + +**Rate Limit Handling Guidance** +- All HTTP clients (e.g., `scraper.py`, `collector.py`) **must** implement centralized rate limit handling to prevent burst violations. +- Use per-session throttling to ensure no more than 5 requests are sent in any 10-second window. +- On receiving HTTP 429 (Too Many Requests), implement exponential backoff and retry logic. Recommended: use the `tenacity` library or custom asyncio-based throttling. +- Example pseudocode: + ```python + import asyncio + from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type + import httpx + + @retry(wait=wait_exponential(multiplier=1, min=2, max=10), stop=stop_after_attempt(5), retry=retry_if_exception_type(httpx.HTTPStatusError)) + async def fetch_with_rate_limit(url, headers): + async with httpx.AsyncClient() as client: + response = await client.get(url, headers=headers) + if response.status_code == 429: + raise httpx.HTTPStatusError("Rate limit exceeded", request=response.request, response=response) + response.raise_for_status() + return response.json() + + # Throttle requests in collector/scraper: + semaphore = asyncio.Semaphore(5) + async def throttled_fetch(*args, **kwargs): + async with semaphore: + return await fetch_with_rate_limit(*args, **kwargs) ## Environment & Build Setup ### Python Version @@ -24,16 +54,28 @@ ### Core Components 1. **app.py** (`src/app.py` - 205 lines): FastAPI application with 30+ endpoints organized by tags (War, Planets, Statistics, Campaigns, Factions, Biomes, System) -2. **scraper.py** (`src/scraper.py` - 115 lines): HTTP client wrapping Hell Divers 2 API (`https://api.live.prod.theadultswim.com/helldivers2`) with methods like `get_war_status()`, `get_planets()`, `get_statistics()` -3. **database.py** (`src/database.py` - 228 lines): SQLite manager with 4 main tables (war_status, statistics, planet_status, campaigns) + indexes for fast historical queries -4. **collector.py** (`src/collector.py` - 97 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle -5. **config.py** (`src/config.py` - 41 lines): Environment-based configuration with DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals) +2. **scraper.py** (`src/scraper.py` - 158 lines): HTTP client for Hell Divers 2 community API with 10 methods: `get_war_status()`, `get_planets()`, `get_statistics()`, `get_factions()`, `get_biomes()`, `get_campaign_info()`, `get_assignments()`, `get_dispatches()`, `get_planet_events()`, `get_planet_status()` +3. **database.py** (`src/database.py` - 360 lines): SQLite manager with 7 tables (war_status, statistics, planet_status, campaigns, assignments, dispatches, planet_events) + indexes for fast queries +4. **collector.py** (`src/collector.py` - 109 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle, collecting all data types with proper error handling +5. **config.py** (`src/config.py` - 57 lines): Environment-based configuration with support for custom API URLs and headers, DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals) ### Lifecycle & Integration - **Startup**: `lifespan()` context manager in app.py starts `collector.start()`, which launches APScheduler -- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 API) → Parser → Database → API endpoints +- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 Community API) → Parser → Database → API endpoints - **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints raise `HTTPException(404/500)` when data unavailable - **JSON Storage**: Complex nested objects stored as TEXT in database with `json.dumps()` for retrieval +- **Configuration**: API base URL is read from `Config.HELLDIVERS_API_BASE` (no hardcoded fallback) +- **Headers**: X-Super-Client and X-Super-Contact headers are automatically added to all API requests. + - **How to configure:** + - The values for these headers are set via config attributes and environment variables: + - `Config.SUPER_CLIENT_HEADER` (from environment variable `SUPER_CLIENT_HEADER`) + - `Config.SUPER_CONTACT_HEADER` (from environment variable `SUPER_CONTACT_HEADER`) + - To set these, add the following to your `.env` file or environment: + ``` + SUPER_CLIENT_HEADER=your-client-name + SUPER_CONTACT_HEADER=your-contact-email-or-identifier + ``` + - These values are loaded in `src/config.py` and injected into all outgoing API requests by the scraper. ## Key Patterns & Conventions @@ -59,8 +101,9 @@ ### Dependencies & Tools Evolution - **Linter Shift**: Replaced `flake8` with `ruff` (faster, more rules, auto-fix support) - **Code Formatting**: Uses `black` (100 char line length) with `ruff format` -- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled -- **Testing**: `pytest` with `pytest-asyncio` for async test support and `pytest-cov` for coverage +- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled; requires `types-requests` for http library stubs +- **Testing**: `pytest` with `pytest-asyncio` for async test support, `pytest-cov` for coverage, and `unittest.mock` for mocking HTTP requests +- **Test Approach**: Unit tests use `@patch()` to mock external API calls, eliminating server dependency and enabling fast CI/CD execution - **Test Output**: Generates `coverage.xml` (for CI upload) and HTML reports - **Coverage Tracking**: Codecov integration via GitHub Actions (non-blocking failures) @@ -71,19 +114,24 @@ 2. Add database storage method in `database.py` using `INSERT INTO` or `UPDATE` with JSON serialization for complex data 3. Create API endpoint in `app.py` with proper tag, docstring, and error handling 4. Add to `collector.py`'s `collect_all_data()` if background collection needed (typical pattern) -5. Test with `demo.py` by adding test functions to the class +5. Test with `demo.py` by adding mocked test functions using `@patch("requests.Session.request")` or patching the exact import path used in `scraper.py` to reliably intercept all HTTP calls -### Modifying Database Schema +### Database Conventions - **Never break existing code**: Create new tables if schema change needed; old tables stay for backward compatibility -- **Always add indexes** on columns used in WHERE or ORDER BY clauses +- **Always add indexes** on columns used in WHERE or ORDER BY clauses (e.g., `CREATE INDEX IF NOT EXISTS idx_assignment_timestamp ON assignments(timestamp)`) - **Use JSON TEXT** for nested/variable data to avoid schema sprawl +- **Auto-incrementing IDs**: `id INTEGER PRIMARY KEY AUTOINCREMENT` on all tables +- **Timestamps**: Every row gets `timestamp DATETIME DEFAULT CURRENT_TIMESTAMP` - **Migration**: No migration system in place; manual database backup recommended before schema changes +- **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT ... ON CONFLICT(unique_col) DO UPDATE SET ...` for idempotency (this preserves row identity and avoids resetting `id`/`timestamp`) ### Adding New Collectors -- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers) +- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers, X-Super-Client/X-Super-Contact headers) +- Add corresponding database save method in `database.py` - Register in `collector.py`'s `collect_all_data()` with try/except and logging - Ensure idempotent: safe to run every 5 minutes without side effects -- Include logging before and after: `logger.info(f"Collected data for X items")` +- Include logging before and after: `logger.info(f"Collected X items")` +- Use `save_*()` methods for data persistence (don't just log collection without storage) ### Dependencies - **Framework**: FastAPI 0.104.1, Uvicorn 0.24.0 @@ -91,6 +139,8 @@ - **HTTP**: Requests 2.31.0 with session reuse and 30-second timeout - **Database**: SQLite3 (built-in); no ORM used—raw SQL with sqlite3 module - **Config**: python-dotenv 1.0.0 for environment variables +- **Type Checking**: types-requests 2.32.4+ for mypy type stubs +- **Testing**: pytest 8.8.2, pytest-asyncio, pytest-cov with unittest.mock for mocking ## Critical Workflows @@ -142,6 +192,8 @@ make run # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p - ❌ **Missing error context**: Log full exceptions with `logger.error(f"...: {e}")` for debugging - ❌ **Hardcoded values**: Use config.py for timeouts (30s), intervals (300s), API URLs - ❌ **Assuming data exists**: Always check `if data:` before processing; scraper methods return `None` on failure +- ❌ **Missing database persistence**: New collectors must call `db.save_*()` methods, not just log +- ❌ **Unsafe nested dict access**: Always check nested keys before accessing (e.g., `campaign.get("planet")` before `campaign["planet"]["index"]`) ## File Map for Quick Reference | Component | File | Purpose | @@ -150,11 +202,11 @@ make run # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p | Project Config | `pyproject.toml` | Build system, dependencies, tool config (ruff, mypy, pytest) | | Makefile | `Makefile` | Bash-based dev/prod/docker/lint targets (aligned with MCP) | | API Routes | `src/app.py` | 30+ FastAPI endpoints with lifecycle management | -| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API | -| Persistence | `src/database.py` | SQLite schema + CRUD operations | +| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API with 10 methods | +| Persistence | `src/database.py` | SQLite schema with 7 tables + CRUD operations | | Scheduling | `src/collector.py` | APScheduler background tasks (5-minute cycle) | | Settings | `src/config.py` | Environment-based configuration classes | -| Integration Tests | `tests/demo.py` | 350+ lines of colored endpoint tests | -| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml | +| Unit Tests | `tests/demo.py` | mocked unit tests (no server dependency) | +| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml, auto-assign.yml | | AI Instructions | `.github/copilot-instructions.md` | This file (aligned with MCP project) | | Documentation | `DEVELOPMENT.md` | Architecture & design decisions |