From d80057dc7afef3a9ded5c84fd47ced1757bcf504 Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 20:34:26 -0700 Subject: [PATCH 1/7] docs: update copilot instructions with API integration and test mocking details --- .github/copilot-instructions.md | 51 ++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ccc4f88..8397514 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,7 +1,12 @@ # High Command API - AI Coding Agent Instructions ## Project Overview -**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, statistics, and faction data via background tasks and serves them through a comprehensive REST API. +**Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, major orders, dispatches, planet events, statistics, and faction data via background tasks and serves them through a comprehensive REST API. + +**API Source**: Community-maintained Hell Divers 2 API at `https://api.helldivers2.dev/api/v1` (257⭐ GitHub, actively maintained) +- Rate limit: 5 requests per 10 seconds +- Required headers: X-Super-Client, X-Super-Contact +- Status: Live and actively maintained with real-time data ## Environment & Build Setup @@ -24,16 +29,18 @@ ### Core Components 1. **app.py** (`src/app.py` - 205 lines): FastAPI application with 30+ endpoints organized by tags (War, Planets, Statistics, Campaigns, Factions, Biomes, System) -2. **scraper.py** (`src/scraper.py` - 115 lines): HTTP client wrapping Hell Divers 2 API (`https://api.live.prod.theadultswim.com/helldivers2`) with methods like `get_war_status()`, `get_planets()`, `get_statistics()` -3. **database.py** (`src/database.py` - 228 lines): SQLite manager with 4 main tables (war_status, statistics, planet_status, campaigns) + indexes for fast historical queries -4. **collector.py** (`src/collector.py` - 97 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle -5. **config.py** (`src/config.py` - 41 lines): Environment-based configuration with DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals) +2. **scraper.py** (`src/scraper.py` - 158 lines): HTTP client for Hell Divers 2 community API with 10 methods: `get_war_status()`, `get_planets()`, `get_statistics()`, `get_factions()`, `get_biomes()`, `get_campaign_info()`, `get_assignments()`, `get_dispatches()`, `get_planet_events()`, `get_planet_status()` +3. **database.py** (`src/database.py` - 360 lines): SQLite manager with 7 tables (war_status, statistics, planet_status, campaigns, assignments, dispatches, planet_events) + indexes for fast queries +4. **collector.py** (`src/collector.py` - 109 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle, collecting all data types with proper error handling +5. **config.py** (`src/config.py` - 57 lines): Environment-based configuration with support for custom API URLs and headers, DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals) ### Lifecycle & Integration - **Startup**: `lifespan()` context manager in app.py starts `collector.start()`, which launches APScheduler -- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 API) → Parser → Database → API endpoints +- **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 Community API) → Parser → Database → API endpoints - **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints raise `HTTPException(404/500)` when data unavailable - **JSON Storage**: Complex nested objects stored as TEXT in database with `json.dumps()` for retrieval +- **Configuration**: API base URL is read from `Config.HELLDIVERS_API_BASE` (no hardcoded fallback) +- **Headers**: X-Super-Client and X-Super-Contact headers automatically added to all API requests ## Key Patterns & Conventions @@ -59,8 +66,9 @@ ### Dependencies & Tools Evolution - **Linter Shift**: Replaced `flake8` with `ruff` (faster, more rules, auto-fix support) - **Code Formatting**: Uses `black` (100 char line length) with `ruff format` -- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled -- **Testing**: `pytest` with `pytest-asyncio` for async test support and `pytest-cov` for coverage +- **Type Checking**: `mypy` (Python 3.9+ compatible) with `ignore_missing_imports` enabled; requires `types-requests` for http library stubs +- **Testing**: `pytest` with `pytest-asyncio` for async test support, `pytest-cov` for coverage, and `unittest.mock` for mocking HTTP requests +- **Test Approach**: Unit tests use `@patch()` to mock external API calls, eliminating server dependency and enabling fast CI/CD execution - **Test Output**: Generates `coverage.xml` (for CI upload) and HTML reports - **Coverage Tracking**: Codecov integration via GitHub Actions (non-blocking failures) @@ -71,19 +79,24 @@ 2. Add database storage method in `database.py` using `INSERT INTO` or `UPDATE` with JSON serialization for complex data 3. Create API endpoint in `app.py` with proper tag, docstring, and error handling 4. Add to `collector.py`'s `collect_all_data()` if background collection needed (typical pattern) -5. Test with `demo.py` by adding test functions to the class +5. Test with `demo.py` by adding mocked test functions using `@patch("requests.get")` or `@patch("requests.post")` -### Modifying Database Schema +### Database Conventions - **Never break existing code**: Create new tables if schema change needed; old tables stay for backward compatibility -- **Always add indexes** on columns used in WHERE or ORDER BY clauses +- **Always add indexes** on columns used in WHERE or ORDER BY clauses (e.g., `CREATE INDEX IF NOT EXISTS idx_assignment_timestamp ON assignments(timestamp)`) - **Use JSON TEXT** for nested/variable data to avoid schema sprawl +- **Auto-incrementing IDs**: `id INTEGER PRIMARY KEY AUTOINCREMENT` on all tables +- **Timestamps**: Every row gets `timestamp DATETIME DEFAULT CURRENT_TIMESTAMP` - **Migration**: No migration system in place; manual database backup recommended before schema changes +- **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT OR REPLACE` for idempotency ### Adding New Collectors -- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers) +- Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers, X-Super-Client/X-Super-Contact headers) +- Add corresponding database save method in `database.py` - Register in `collector.py`'s `collect_all_data()` with try/except and logging - Ensure idempotent: safe to run every 5 minutes without side effects -- Include logging before and after: `logger.info(f"Collected data for X items")` +- Include logging before and after: `logger.info(f"Collected X items")` +- Use `save_*()` methods for data persistence (don't just log collection without storage) ### Dependencies - **Framework**: FastAPI 0.104.1, Uvicorn 0.24.0 @@ -91,6 +104,8 @@ - **HTTP**: Requests 2.31.0 with session reuse and 30-second timeout - **Database**: SQLite3 (built-in); no ORM used—raw SQL with sqlite3 module - **Config**: python-dotenv 1.0.0 for environment variables +- **Type Checking**: types-requests 2.32.4+ for mypy type stubs +- **Testing**: pytest 8.8.2, pytest-asyncio, pytest-cov with unittest.mock for mocking ## Critical Workflows @@ -142,6 +157,8 @@ make run # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p - ❌ **Missing error context**: Log full exceptions with `logger.error(f"...: {e}")` for debugging - ❌ **Hardcoded values**: Use config.py for timeouts (30s), intervals (300s), API URLs - ❌ **Assuming data exists**: Always check `if data:` before processing; scraper methods return `None` on failure +- ❌ **Missing database persistence**: New collectors must call `db.save_*()` methods, not just log +- ❌ **Unsafe nested dict access**: Always check nested keys before accessing (e.g., `campaign.get("planet")` before `campaign["planet"]["index"]`) ## File Map for Quick Reference | Component | File | Purpose | @@ -150,11 +167,11 @@ make run # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p | Project Config | `pyproject.toml` | Build system, dependencies, tool config (ruff, mypy, pytest) | | Makefile | `Makefile` | Bash-based dev/prod/docker/lint targets (aligned with MCP) | | API Routes | `src/app.py` | 30+ FastAPI endpoints with lifecycle management | -| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API | -| Persistence | `src/database.py` | SQLite schema + CRUD operations | +| Data Fetching | `src/scraper.py` | HTTP client for Hell Divers 2 API with 10 methods | +| Persistence | `src/database.py` | SQLite schema with 7 tables + CRUD operations | | Scheduling | `src/collector.py` | APScheduler background tasks (5-minute cycle) | | Settings | `src/config.py` | Environment-based configuration classes | -| Integration Tests | `tests/demo.py` | 350+ lines of colored endpoint tests | -| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml | +| Unit Tests | `tests/demo.py` | 11 mocked unit tests (no server dependency) | +| CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml, auto-assign.yml | | AI Instructions | `.github/copilot-instructions.md` | This file (aligned with MCP project) | | Documentation | `DEVELOPMENT.md` | Architecture & design decisions | From 53573eb121f1a5bfac7753ecb2186206c181253d Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 20:41:38 -0700 Subject: [PATCH 2/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8397514..f35a069 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -3,7 +3,7 @@ ## Project Overview **Hell Divers 2 API**: Production-ready FastAPI application for real-time scraping and tracking of Hell Divers 2 game data. The system continuously collects war status, planet information, campaigns, major orders, dispatches, planet events, statistics, and faction data via background tasks and serves them through a comprehensive REST API. -**API Source**: Community-maintained Hell Divers 2 API at `https://api.helldivers2.dev/api/v1` (257⭐ GitHub, actively maintained) +**API Source**: Community-maintained Hell Divers 2 API at `https://api.helldivers2.dev/api/v1` (popular, actively maintained) - Rate limit: 5 requests per 10 seconds - Required headers: X-Super-Client, X-Super-Contact - Status: Live and actively maintained with real-time data From 3c2fcc35ffe7af6bbf91d4c78ea8b0b06c257677 Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 20:41:59 -0700 Subject: [PATCH 3/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index f35a069..a28749a 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -79,7 +79,7 @@ 2. Add database storage method in `database.py` using `INSERT INTO` or `UPDATE` with JSON serialization for complex data 3. Create API endpoint in `app.py` with proper tag, docstring, and error handling 4. Add to `collector.py`'s `collect_all_data()` if background collection needed (typical pattern) -5. Test with `demo.py` by adding mocked test functions using `@patch("requests.get")` or `@patch("requests.post")` +5. Test with `demo.py` by adding mocked test functions using `@patch("requests.Session.request")` or patching the exact import path used in `scraper.py` to reliably intercept all HTTP calls ### Database Conventions - **Never break existing code**: Create new tables if schema change needed; old tables stay for backward compatibility From 79972e9dce8c7bea47abec881ad096ce6fd91b23 Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 20:42:14 -0700 Subject: [PATCH 4/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index a28749a..8c8dcf1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -88,7 +88,7 @@ - **Auto-incrementing IDs**: `id INTEGER PRIMARY KEY AUTOINCREMENT` on all tables - **Timestamps**: Every row gets `timestamp DATETIME DEFAULT CURRENT_TIMESTAMP` - **Migration**: No migration system in place; manual database backup recommended before schema changes -- **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT OR REPLACE` for idempotency +- **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT ... ON CONFLICT(unique_col) DO UPDATE SET ...` for idempotency (this preserves row identity and avoids resetting `id`/`timestamp`) ### Adding New Collectors - Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers, X-Super-Client/X-Super-Contact headers) From 22fad5b8549d20796216f38f9d9f638cc6b4d52e Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 20:42:25 -0700 Subject: [PATCH 5/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8c8dcf1..951ff0e 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -171,7 +171,7 @@ make run # Runs: python -m uvicorn src.app:app --host 0.0.0.0 --p | Persistence | `src/database.py` | SQLite schema with 7 tables + CRUD operations | | Scheduling | `src/collector.py` | APScheduler background tasks (5-minute cycle) | | Settings | `src/config.py` | Environment-based configuration classes | -| Unit Tests | `tests/demo.py` | 11 mocked unit tests (no server dependency) | +| Unit Tests | `tests/demo.py` | mocked unit tests (no server dependency) | | CI/CD Workflows | `.github/workflows/` | tests.yml, docker.yml, auto-approve.yml, auto-assign.yml | | AI Instructions | `.github/copilot-instructions.md` | This file (aligned with MCP project) | | Documentation | `DEVELOPMENT.md` | Architecture & design decisions | From 4d76a6fa1346e5a190237badca43b8d249b116e8 Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 21:01:01 -0700 Subject: [PATCH 6/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 951ff0e..2407c8d 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -8,6 +8,31 @@ - Required headers: X-Super-Client, X-Super-Contact - Status: Live and actively maintained with real-time data + +**Rate Limit Handling Guidance** +- All HTTP clients (e.g., `scraper.py`, `collector.py`) **must** implement centralized rate limit handling to prevent burst violations. +- Use per-session throttling to ensure no more than 5 requests are sent in any 10-second window. +- On receiving HTTP 429 (Too Many Requests), implement exponential backoff and retry logic. Recommended: use the `tenacity` library or custom asyncio-based throttling. +- Example pseudocode: + ```python + import asyncio + from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type + import httpx + + @retry(wait=wait_exponential(multiplier=1, min=2, max=10), stop=stop_after_attempt(5), retry=retry_if_exception_type(httpx.HTTPStatusError)) + async def fetch_with_rate_limit(url, headers): + async with httpx.AsyncClient() as client: + response = await client.get(url, headers=headers) + if response.status_code == 429: + raise httpx.HTTPStatusError("Rate limit exceeded", request=response.request, response=response) + response.raise_for_status() + return response.json() + + # Throttle requests in collector/scraper: + semaphore = asyncio.Semaphore(5) + async def throttled_fetch(*args, **kwargs): + async with semaphore: + return await fetch_with_rate_limit(*args, **kwargs) ## Environment & Build Setup ### Python Version From e78b63d091eaa81ff96b8b3c01acc940ee08113a Mon Sep 17 00:00:00 2001 From: Lee Chapman Date: Sun, 19 Oct 2025 21:01:19 -0700 Subject: [PATCH 7/7] Update .github/copilot-instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/copilot-instructions.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 2407c8d..799fd98 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -65,7 +65,17 @@ - **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints raise `HTTPException(404/500)` when data unavailable - **JSON Storage**: Complex nested objects stored as TEXT in database with `json.dumps()` for retrieval - **Configuration**: API base URL is read from `Config.HELLDIVERS_API_BASE` (no hardcoded fallback) -- **Headers**: X-Super-Client and X-Super-Contact headers automatically added to all API requests +- **Headers**: X-Super-Client and X-Super-Contact headers are automatically added to all API requests. + - **How to configure:** + - The values for these headers are set via config attributes and environment variables: + - `Config.SUPER_CLIENT_HEADER` (from environment variable `SUPER_CLIENT_HEADER`) + - `Config.SUPER_CONTACT_HEADER` (from environment variable `SUPER_CONTACT_HEADER`) + - To set these, add the following to your `.env` file or environment: + ``` + SUPER_CLIENT_HEADER=your-client-name + SUPER_CONTACT_HEADER=your-contact-email-or-identifier + ``` + - These values are loaded in `src/config.py` and injected into all outgoing API requests by the scraper. ## Key Patterns & Conventions