Documentation update: Cache fallback feature and system_status table

.github/copilot-instructions.md

@@ -53,16 +53,18 @@
 - **`make check-all`** - Format, lint, test (complete quality gate for PRs)
 
 ### Core Components
-1. **app.py** (`src/app.py` - 205 lines): FastAPI application with 30+ endpoints organized by tags (War, Planets, Statistics, Campaigns, Factions, Biomes, System)
+1. **app.py** (`src/app.py` - 230 lines): FastAPI application with 30+ endpoints organized by tags (War, Planets, Statistics, Campaigns, Factions, Biomes, System). Implements cache fallback pattern for critical endpoints.
 2. **scraper.py** (`src/scraper.py` - 175 lines): HTTP client for Hell Divers 2 community API with rate limiting (2 second delay between requests to stay within 5 requests/10 seconds limit). 10 methods: `get_war_status()`, `get_planets()`, `get_statistics()`, `get_factions()`, `get_biomes()`, `get_campaign_info()`, `get_assignments()`, `get_dispatches()`, `get_planet_events()`, `get_planet_status()`
-3. **database.py** (`src/database.py` - 360 lines): SQLite manager with 7 tables (war_status, statistics, planet_status, campaigns, assignments, dispatches, planet_events) + indexes for fast queries
-4. **collector.py** (`src/collector.py` - 109 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle, collecting all data types with proper error handling
+3. **database.py** (`src/database.py` - 525 lines): SQLite manager with 8 tables (war_status, statistics, planet_status, campaigns, assignments, dispatches, planet_events, system_status) + indexes for fast queries. Includes snapshot methods for cache fallback.
+4. **collector.py** (`src/collector.py` - 145 lines): APScheduler-based background task runner that calls `collect_all_data()` every 5 minutes (300s) during app lifecycle, collecting all data types with proper error handling. Updates system_status table with upstream API availability.
 5. **config.py** (`src/config.py` - 57 lines): Environment-based configuration with support for custom API URLs and headers, DevelopmentConfig (1-minute intervals) vs ProductionConfig (5-minute intervals)
 
 ### Lifecycle & Integration
 - **Startup**: `lifespan()` context manager in app.py starts `collector.start()`, which launches APScheduler
 - **Data Flow**: Collector → Scraper → (HTTP to Hell Divers 2 Community API) → Parser → Database → API endpoints
-- **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints raise `HTTPException(404/500)` when data unavailable
+- **Error Handling**: Scraper returns `None` on failures with logged exceptions; endpoints with cache fallback try cached data before raising `HTTPException(503)`, other endpoints raise `HTTPException(404/500)` when data unavailable
+- **Cache Fallback**: Critical endpoints (`/api/campaigns`, `/api/planets`, `/api/planets/{planet_index}`, `/api/factions`, `/api/biomes`) automatically fall back to cached data when upstream API fails, returning 503 only when both live fetch AND cache fail
+- **System Status Tracking**: Collector updates `system_status` table with `upstream_api_available` boolean after each collection cycle (success=True, failure=False)
 - **JSON Storage**: Complex nested objects stored as TEXT in database with `json.dumps()` for retrieval
 - **Configuration**: API base URL is read from `Config.HELLDIVERS_API_BASE` (no hardcoded fallback)
 - **Headers**: X-Super-Client and X-Super-Contact headers are automatically added to all API requests.
@@ -130,8 +132,9 @@
 - **Use JSON TEXT** for nested/variable data to avoid schema sprawl
 - **Auto-incrementing IDs**: `id INTEGER PRIMARY KEY AUTOINCREMENT` on all tables
 - **Timestamps**: Every row gets `timestamp DATETIME DEFAULT CURRENT_TIMESTAMP`
-- **Migration**: No migration system in place; manual database backup recommended before schema changes
+- **Migration**: No migration system in place; manual database backup recommended before schema changes. New tables are auto-created via `CREATE TABLE IF NOT EXISTS`.
 - **Unique constraints**: Use `UNIQUE` for business logic keys (e.g., campaign_id, assignment_id) with `INSERT ... ON CONFLICT(unique_col) DO UPDATE SET ...` for idempotency (this preserves row identity and avoids resetting `id`/`timestamp`)
+- **System metadata**: Use `system_status` table for internal tracking (e.g., upstream_api_available) with `INSERT OR REPLACE` pattern for upserts
 
 ### Adding New Collectors
 - Add method to `scraper.py` following existing patterns (session timeout, User-Agent headers, X-Super-Client/X-Super-Contact headers)
@@ -141,6 +144,16 @@
 - Include logging before and after: `logger.info(f"Collected X items")`
 - Use `save_*()` methods for data persistence (don't just log collection without storage)
 
+### Adding Cache Fallback to Endpoints
+For critical endpoints that should remain available during upstream outages:
+1. Try live API fetch first: `data = scraper.get_resource()`
+2. Check for failure: `if data is None:`
+3. Fall back to cache: `data = db.get_latest_resource_snapshot()`
+4. Return data if available: `if data is not None: return data`
+5. Raise 503 if both fail: `raise HTTPException(status_code=503, detail="...and no cached data")`
+6. Implement corresponding `get_latest_*_snapshot()` method in database.py that retrieves most recent cached data
+7. Pattern example: See `/api/campaigns`, `/api/planets`, `/api/factions` in app.py
+
 ### Dependencies
 - **Framework**: FastAPI 0.104.1, Uvicorn 0.24.0
 - **Scheduler**: APScheduler 3.10.4 (no concurrent execution; uses background threads)

.gitignore

@@ -104,6 +104,13 @@ log/
 *.bak
 *.swp~
 
+# ============================================================================
+# Development Summary Files (auto-generated during development)
+# ============================================================================
+*_SUMMARY.md
+COMPLETE_GUIDE.md
+WORKFLOW_TEST_RESULTS.md
+
 # ============================================================================
 # OS-Specific
 # ============================================================================

DEVELOPMENT.md

@@ -54,10 +54,17 @@
   - `statistics`: Global game stats
   - `planet_status`: Individual planet data
   - `campaigns`: Campaign information
+  - `assignments`: Major orders
+  - `dispatches`: News and announcements
+  - `planet_events`: Special planet events
+  - `system_status`: Internal metadata (upstream API status)
 - **Key Methods**:
   - `save_*()`: Insert data
   - `get_*()`: Retrieve data
   - `*_history()`: Get historical data
+  - `get_latest_*_snapshot()`: Get cached data for fallback
+  - `set_upstream_status()`: Track upstream API availability
+  - `get_upstream_status()`: Check last known upstream status
 
 ### `collector.py` - Background Tasks
 - **Purpose**: Schedule and manage data collection
@@ -97,6 +104,22 @@ All responses follow a consistent JSON format:
 }
 ```
 
+### Cache Fallback Behavior
+
+Certain endpoints implement cache fallback for resilience:
+- **Endpoints with cache fallback**: `/api/campaigns`, `/api/planets`, `/api/planets/{planet_index}`, `/api/factions`, `/api/biomes`
+- **Fallback logic**:
+  1. Try to fetch from live upstream API
+  2. If upstream fails, retrieve from database cache
+  3. If cache exists, return 200 with cached data
+  4. If no cache exists, return 503 Service Unavailable
+
+**Status Codes**:
+- `200 OK`: Success (may be live or cached data)
+- `404 Not Found`: Resource doesn't exist or no data collected
+- `503 Service Unavailable`: Upstream API down AND no cached data available
+- `500 Internal Server Error`: Unexpected server error
+
 ### Endpoint Patterns
 
 ```
@@ -158,6 +181,42 @@ async def get_new_data():
     raise HTTPException(status_code=404, detail="No data available")
 ```
 
+### 4. (Optional) Add Cache Fallback
+
+For critical endpoints that should continue serving during upstream outages:
+```python
+@app.get("/api/new-data", tags=["New Data"])
+async def get_new_data():
+    """Get new data (with cache fallback)"""
+    # Try live API first
+    data = scraper.get_new_data()
+
+    # Fallback to cache if live API fails
+    if data is None:
+        data = db.get_latest_new_data_snapshot()
+
+    if data is not None:
+        return data
+    raise HTTPException(status_code=503, detail="No new data available (live fetch failed and no cached data)")
+```
+
+### System Status Tracking
+
+The `system_status` table tracks internal metadata:
+- **Purpose**: Monitor upstream API health and other system metrics
+- **Usage**: Collector updates upstream status after each collection cycle
+- **Schema**:
+  ```sql
+  CREATE TABLE system_status (
+      id INTEGER PRIMARY KEY AUTOINCREMENT,
+      status_key TEXT UNIQUE,
+      status_value BOOLEAN,
+      timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
+  )
+  ```
+- **Current Keys**:
+  - `upstream_api_available`: Boolean indicating if Hell Divers 2 API is reachable
+
 ## Testing
 
 ### Manual Testing
@@ -289,6 +348,31 @@ lsof -i :5000
 pip install -r requirements.txt --force-reinstall
 ```
 
+### Issue: "503 errors on cache-fallback endpoints"
+**Solution**: 
+- Check if upstream API is accessible: `curl https://api.helldivers2.dev/api/v1/war/info`
+- Verify database has cached data: `sqlite3 helldivers2.db "SELECT COUNT(*) FROM planet_status"`
+- Wait for collector to run and populate cache (runs every 5 minutes)
+
+## Database Migration Notes
+
+### Adding system_status Table (Cache Fallback Feature)
+
+If upgrading from a version before the cache-fallback feature, the `system_status` table will be automatically created on next startup. No manual migration is required.
+
+**What changed**:
+- New table: `system_status` with columns `id`, `status_key`, `status_value`, `timestamp`
+- New index: `idx_system_status_key` on `status_key` column
+- New methods: `set_upstream_status()`, `get_upstream_status()` in `database.py`
+- New snapshot methods: `get_latest_planets_snapshot()`, `get_latest_campaigns_snapshot()`, etc.
+
+**To verify migration**:
+```bash
+sqlite3 helldivers2.db "SELECT name FROM sqlite_master WHERE type='table' AND name='system_status'"
+```
+
+Expected output: `system_status`
+
 ## Code Style
 
 - Use type hints for all functions

README.md

@@ -11,6 +11,7 @@ High Command API provides comprehensive access to Hell Divers 2 game statistics,
 - ✅ **Real-time Data Collection**: Automatic background scraping every 5 minutes
 - ✅ **Comprehensive Endpoints**: 30+ endpoints for accessing game data
 - ✅ **Historical Tracking**: SQLite database with optimized queries
+- ✅ **Cache Fallback**: Graceful degradation when upstream API is unavailable
 - ✅ **Interactive Documentation**: Auto-generated Swagger UI and ReDoc
 - ✅ **Production Ready**: Docker support, error handling, type hints
 - ✅ **Developer Friendly**: Clean code, extensive documentation
@@ -30,10 +31,10 @@ high-command-api/
 │   ├── __init__.py
 │   └── demo.py            # Comprehensive tests
 ├── docs/                   # Documentation
-│   ├── DEPLOYMENT.md      # Production deployment guide
-│   ├── DEVELOPMENT.md     # Architecture guide
+│   ├── API.md             # API reference
+│   ├── ARCHITECTURE.md    # Architecture & design
 │   ├── QUICKREF.md        # Quick reference
-│   └── API.md             # API reference
+│   └── DEPLOYMENT.md      # Deployment guide
 ├── Makefile               # Project automation
 ├── Dockerfile             # Container image
 ├── docker-compose.yml     # Multi-container setup
@@ -42,6 +43,17 @@ high-command-api/
 └── pyproject.toml        # Project metadata
 ```
 
+**Note**: Files like `*_SUMMARY.md`, `COMPLETE_GUIDE.md`, etc. are development artifacts and can be safely ignored.
+
+## Documentation
+
+- **[README.md](README.md)** - This file; project overview and quick start
+- **[docs/API.md](docs/API.md)** - Complete API endpoint reference with examples
+- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** - System architecture and database schema
+- **[docs/DEVELOPMENT.md](DEVELOPMENT.md)** - Development guide and patterns
+- **[docs/DEPLOYMENT.md](DEPLOYMENT.md)** - Production deployment instructions
+- **[.github/copilot-instructions.md](.github/copilot-instructions.md)** - AI coding agent instructions
+
 ## Installation
 
 ### Prerequisites
@@ -152,9 +164,24 @@ The API uses SQLite to store:
 - **statistics**: Game-wide statistics with timestamps
 - **planet_status**: Individual planet status snapshots
 - **campaigns**: Active and completed campaigns
+- **assignments**: Major orders and assignments
+- **dispatches**: News and announcements
+- **planet_events**: Special events on planets
+- **system_status**: Internal metadata (e.g., upstream API availability)
 
 All data includes timestamps for historical tracking and analysis.
 
+### Cache Fallback Strategy
+
+When the upstream Hell Divers 2 API is unavailable, the following endpoints automatically fall back to cached data:
+- `/api/campaigns` - Returns most recent campaign snapshot
+- `/api/planets` - Returns most recent planets snapshot
+- `/api/planets/{planet_index}` - Returns most recent planet status
+- `/api/factions` - Returns factions from latest war status
+- `/api/biomes` - Returns biomes from latest planet data
+
+These endpoints return **503 Service Unavailable** only when both the live API fails AND no cached data is available. This ensures maximum uptime and data availability.
+
 ## Background Data Collection
 
 The application automatically collects data on a configurable interval (default 5 minutes):
@@ -229,12 +256,13 @@ MIT
 
 Contributions are welcome! Please feel free to submit pull requests.
 
-## Troubleshooting
+### Troubleshooting
 
 ### No data available
-- Check that the API can reach `https://api.live.prod.theadultswim.com`
+- Check that the API can reach `https://api.helldivers2.dev`
 - Verify network connectivity and firewall settings
 - Check logs for specific error messages
+- If upstream API is down, endpoints with cache fallback will serve cached data (with 503 status if no cache exists)
 
 ### Database errors
 - Ensure write permissions to the database directory
@@ -243,3 +271,8 @@ Contributions are welcome! Please feel free to submit pull requests.
 ### Scheduler not running
 - Check logs for background scheduler errors
 - Verify APScheduler is properly installed
+
+### Understanding Error Codes
+- **404 Not Found**: Resource doesn't exist or no data has been collected yet
+- **503 Service Unavailable**: Upstream API is down and no cached data is available (for endpoints with cache fallback)
+- **500 Internal Server Error**: Unexpected server error occurred