docs: update copilot instructions with API integration details

.github/copilot-instructions.md

@@ -1,8 +1,38 @@
 # High Command API - AI Coding Agent Instructions
 
 ## Project Overview
-**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, statistics, and faction data via background tasks and serves them through a comprehensive REST API.
-
+**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, major orders, dispatches, planet events, statistics, and faction data via background tasks and serves them through a comprehensive REST API.
+
+**API Source**: Community-maintained Hell Divers 2 API at `https://api.helldivers2.dev/api/v1` (popular, actively maintained)
+- Rate limit: 5 requests per 10 seconds
+- Required headers: X-Super-Client, X-Super-Contact
+- Status: Live and actively maintained with real-time data
+
+
+**Rate Limit Handling Guidance**
+- All HTTP clients (e.g., `scraper.py`, `collector.py`) **must** implement centralized rate limit handling to prevent burst violations.
+- Use per-session throttling to ensure no more than 5 requests are sent in any 10-second window.
+- On receiving HTTP 429 (Too Many Requests), implement exponential backoff and retry logic. Recommended: use the `tenacity` library or custom asyncio-based throttling.
+- Example pseudocode:
+  ```python
+  import asyncio
+  from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type
+  import httpx
+
+  @retry(wait=wait_exponential(multiplier=1, min=2, max=10), stop=stop_after_attempt(5), retry=retry_if_exception_type(httpx.HTTPStatusError))
+  async def fetch_with_rate_limit(url, headers):
+      async with httpx.AsyncClient() as client:
+          response = await client.get(url, headers=headers)
+          if response.status_code == 429:
+              raise httpx.HTTPStatusError("Rate limit exceeded", request=response.request, response=response)
+          response.raise_for_status()
+          return response.json()
+
+  # Throttle requests in collector/scraper:
+  semaphore = asyncio.Semaphore(5)
+  async def throttled_fetch(*args, **kwargs):
+      async with semaphore:
+          return await fetch_with_rate_limit(*args, **kwargs)
 ## Environment & Build Setup
 
 ### Python Version
@@ -24,16 +54,28 @@
 
 ### Core Components
 1. **app.py** (`src/app.py` - 205 lines): FastAPI application with 30+ endpoints organized by tags (War, Planets, Statistics, Campaigns, Factions, Biomes, System)
-2. **scraper.py** (`src/scraper.py` - 115 lines): HTTP client wrapping Hell Divers 2 API (`https://api.live.prod.theadultswim.com/helldivers2`) with methods like `get_war_status()`, `get_planets()`, `get_statistics()`
-3. **database.py** (`src/database.py` - 228 lines): SQLite manager with 4 main tables (war_status, statistics, planet_status, campaigns) + indexes for fast historical queries
-4. **collector.py** (`src/collector.py` - 97 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle
-5. **config.py** (`src/config.py` - 41 lines): Environment-based configuration with DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals)
+2. **scraper.py** (`src/scraper.py` - 158 lines): HTTP client for Hell Divers 2 community API with 10 methods: `get_war_status()`, `get_planets()`, `get_statistics()`, `get_factions()`, `get_biomes()`, `get_campaign_info()`, `get_assignments()`, `get_dispatches()`, `get_planet_events()`, `get_planet_status()`
+3. **database.py** (`src/database.py` - 360 lines): SQLite manager with 7 tables (war_status, statistics, planet_status, campaigns, assignments, dispatches, planet_events) + indexes for fast queries
+4. **collector.py** (`src/collector.py` - 109 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle, collecting all data types with proper error handling
+5. **config.py** (`src/config.py` - 57 lines): Environment-based configuration with support for custom API URLs and headers, DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals)
 
 ### Lifecycle & Integration
 - **Startup**: `lifespan()` context manager in app.py starts `collector.start()`, which launches APScheduler
-- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 API) → Parser → Database → API endpoints
+- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 Community API) → Parser → Database → API endpoints
 - **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints raise `HTTPException(404/500)` when data unavailable
 - **JSON Storage**: Complex nested objects stored as TEXT in database with `json.dumps()` for retrieval
+- **Configuration**: API base URL is read from `Config.HELLDIVERS_API_BASE` (no hardcoded fallback)
+- **Headers**: X-Super-Client and X-Super-Contact headers are automatically added to all API requests.
+  - **How to configure:**  
+    - The values for these headers are set via config attributes and environment variables:  
+      - `Config.SUPER_CLIENT_HEADER` (from environment variable `SUPER_CLIENT_HEADER`)  
+      - `Config.SUPER_CONTACT_HEADER` (from environment variable `SUPER_CONTACT_HEADER`)  
+    - To set these, add the following to your `.env` file or environment:  
+      ```
+      SUPER_CLIENT_HEADER=your-client-name
+      SUPER_CONTACT_HEADER=your-contact-email-or-identifier
+      ```
+    - These values are loaded in `src/config.py` and injected into all outgoing API requests by the scraper.
 
 ## Key Patterns & Conventions
 
@@ -59,8 +101,9 @@
 ### Dependencies & Tools Evolution
 - **Linter Shift**: Replaced `flake8` with `ruff` (faster, more rules, auto-fix support)
 - **Code Formatting**: Uses `black` (100 char line length) with `ruff format`
-- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled
-- **Testing**: `pytest` with `pytest-asyncio` for async test support and `pytest-cov` for coverage
+- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled; requires `types-requests` for http library stubs
+- **Testing**: `pytest` with `pytest-asyncio` for async test support, `pytest-cov` for coverage, and `unittest.mock` for mocking HTTP requests
+- **Test Approach**: Unit tests use `@patch()` to mock external API calls, eliminating server dependency and enabling fast CI/CD execution
 - **Test Output**: Generates `coverage.xml` (for CI upload) and HTML reports
 - **Coverage Tracking**: Codecov integration via GitHub Actions (non-blocking failures)
 
@@ -71,26 +114,33 @@
 2. Add database storage method in `database.py` using `INSERT INTO` or `UPDATE` with JSON serialization for complex data
 3. Create API endpoint in `app.py` with proper tag, docstring, and error handling
 4. Add to `collector.py`'s `collect_all_data()` if background collection needed (typical pattern)
-5. Test with `demo.py` by adding test functions to the class
+5. Test with `demo.py` by adding mocked test functions using `@patch("requests.Session.request")` or patching the exact import path used in `scraper.py` to reliably intercept all HTTP calls
 
-### Modifying Database Schema
+### Database Conventions
 - **Never break existing code**: Create new tables if schema change needed; old tables stay for backward compatibility
-- **Always add indexes** on columns used in WHERE or ORDER BY clauses
+- **Always add indexes** on columns used in WHERE or ORDER BY clauses (e.g., `CREATE INDEX IF NOT EXISTS idx_assignment_timestamp ON assignments(timestamp)`)
 - **Use JSON TEXT** for nested/variable data to avoid schema sprawl
+- **Auto-incrementing IDs**: `id INTEGER PRIMARY KEY AUTOINCREMENT` on all tables
+- **Timestamps**: Every row gets `timestamp DATETIME DEFAULT CURRENT_TIMESTAMP`
 - **Migration**: No migration system in place; manual database backup recommended before schema changes
+- **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT ... ON CONFLICT(unique_col) DO UPDATE SET ...` for idempotency (this preserves row identity and avoids resetting `id`/`timestamp`)
 
 ### Adding New Collectors
-- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers)
+- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers, X-Super-Client/X-Super-Contact headers)
+- Add corresponding database save method in `database.py` 
 - Register in `collector.py`'s `collect_all_data()` with try/except and logging
 - Ensure idempotent: safe to run every 5 minutes without side effects
-- Include logging before and after: `logger.info(f"Collected data for X items")`
+- Include logging before and after: `logger.info(f"Collected X items")`
+- Use `save_*()` methods for data persistence (don't just log collection without storage)
 
 ### Dependencies
 - **Framework**: FastAPI 0.104.1, Uvicorn 0.24.0
 - **Scheduler**: APScheduler 3.10.4 (no concurrent execution; uses background threads)
 - **HTTP**: Requests 2.31.0 with session reuse and 30-second timeout
 - **Database**: SQLite3 (built-in); no ORM used—raw SQL with sqlite3 module
 - **Config**: python-dotenv 1.0.0 for environment variables
+- **Type Checking**: types-requests 2.32.4+ for mypy type stubs
+- **Testing**: pytest 8.8.2, pytest-asyncio, pytest-cov with unittest.mock for mocking
 
 ## Critical Workflows
 
@@ -142,6 +192,8 @@ make run                # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p
 - ❌ **Missing error context**: Log full exceptions with `logger.error(f"...: {e}")` for debugging
 - ❌ **Hardcoded values**: Use config.py for timeouts (30s), intervals (300s), API URLs
 - ❌ **Assuming data exists**: Always check `if data:` before processing; scraper methods return `None` on failure
+- ❌ **Missing database persistence**: New collectors must call `db.save_*()` methods, not just log
+- ❌ **Unsafe nested dict access**: Always check nested keys before accessing (e.g., `campaign.get("planet")` before `campaign["planet"]["index"]`)
 
 ## File Map for Quick Reference
 | Component | File | Purpose |
@@ -150,11 +202,11 @@ make run                # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p
 | Project Config | `pyproject.toml` | Build system, dependencies, tool config (ruff, mypy, pytest) |
 | Makefile | `Makefile` | Bash-based dev/prod/docker/lint targets (aligned with MCP) |
 | API Routes | `src/app.py` | 30+ FastAPI endpoints with lifecycle management |
-| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API |
-| Persistence | `src/database.py` | SQLite schema + CRUD operations |
+| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API with 10 methods |
+| Persistence | `src/database.py` | SQLite schema with 7 tables + CRUD operations |
 | Scheduling | `src/collector.py` | APScheduler background tasks (5-minute cycle) |
 | Settings | `src/config.py` | Environment-based configuration classes |
-| Integration Tests | `tests/demo.py` | 350+ lines of colored endpoint tests |
-| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml |
+| Unit Tests | `tests/demo.py` | mocked unit tests (no server dependency) |
+| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml, auto-assign.yml |
 | AI Instructions | `.github/copilot-instructions.md` | This file (aligned with MCP project) |
 | Documentation | `DEVELOPMENT.md` | Architecture & design decisions |