diff --git a/docs/BACKEND_DATA_REFRESH.md b/docs/BACKEND_DATA_REFRESH.md new file mode 100644 index 0000000..107a8c7 --- /dev/null +++ b/docs/BACKEND_DATA_REFRESH.md @@ -0,0 +1,246 @@ +# Backend Data Refresh Guide + +## Issue Diagnosis ✅ + +The API is working correctly, but the backend database only contains test/stale data: + +- ✅ Sorting logic: Working correctly (newest first) +- ✅ Timestamp parsing: Working correctly +- ✅ API response format: Correct +- ❌ **Data freshness**: Only has ~10 dispatches from March 2024 +- ❌ **No current data**: No October 2025 dispatches in database + +## Root Causes + +This could happen due to: + +1. **Scraper not running** - Background collector/scraper task isn't active +2. **Upstream API unavailable** - Hell Divers 2 API down or rate-limited +3. **Test data only** - Development database with fixtures, not live data +4. **Initial state** - First time running, scraper hasn't collected data yet + +## Solutions + +### Solution 1: Manually Trigger Data Refresh (Quickest) + +Use the refresh endpoints to force data collection: + +```bash +# Refresh all dispatches +curl -X POST http://localhost:5000/api/dispatches/refresh + +# Refresh assignments +curl -X POST http://localhost:5000/api/assignments/refresh + +# Refresh planet events +curl -X POST http://localhost:5000/api/planet-events/refresh + +# Refresh all war data +curl -X POST http://localhost:5000/api/war/status/refresh +``` + +Expected response: +```json +{ + "success": true, + "data": { + "dispatches": [ + { + "id": 1, + "message": "...", + "published": "2025-10-21T..." + } + ] + } +} +``` + +### Solution 2: Start Background Collector (Automatic Updates) + +Ensure the collector is running when API starts: + +```bash +# Check collector status +curl http://localhost:5000/api/health + +# Expected response includes: +# "collector_running": true +``` + +If collector is not running, the API is missing the background refresh loop. + +### Solution 3: Verify Upstream API Connection + +Test connectivity to Hell Divers 2 community API: + +```bash +# Direct test to Hell Divers 2 API (no rate limiting) +curl -H "X-Super-Client: high-command-test" \ + -H "X-Super-Contact: dev@example.com" \ + https://api.helldivers2.dev/api/v1/dispatches +``` + +If this returns fresh data, the upstream is working. + +### Solution 4: Reset and Reinitialize Database (Nuclear Option) + +If all else fails, remove old database and force fresh collection: + +```bash +# Stop the API first (Ctrl+C in terminal) + +# Backup old database +cp helldivers2.db helldivers2.db.backup + +# Remove old database +rm helldivers2.db + +# Restart API - will create fresh database and collect data +make dev +``` + +## Data Collection Flow + +``` +Hell Divers 2 Community API + ↓ +Scraper (src/scraper.py) + ↓ +Collector (src/collector.py) - runs every 5 minutes + ↓ +Database (helldivers2.db - SQLite) + ↓ +FastAPI Endpoints (src/app.py) + ↓ +Frontend/Client +``` + +## Configuration + +### Check Collector Settings (src/config.py) + +```python +# Development: collects every 60 seconds +class DevelopmentConfig: + COLLECTION_INTERVAL = 60 + +# Production: collects every 300 seconds (5 minutes) +class ProductionConfig: + COLLECTION_INTERVAL = 300 +``` + +### Enable Debug Logging + +Check logs to see if collector is working: + +```bash +# In development, watch for: +# "Collector started" +# "Collecting all game data..." +# "Collected X dispatches" +``` + +## Verification Checklist + +- [ ] API starts without errors +- [ ] Health endpoint shows `collector_running: true` +- [ ] Manual refresh endpoints return data (200 status) +- [ ] Database file `helldivers2.db` exists and has size > 100KB +- [ ] New dispatches appear after running refresh +- [ ] Timestamps are recent (not March 2024) + +## Expected Behavior After Fix + +After ensuring fresh data collection: + +```bash +# Should return current October 2025 dispatches +curl http://localhost:5000/api/dispatches?limit=10&sort=newest + +# Response should have: +# - Multiple dispatches +# - Recent timestamps (October 2025) +# - Properly formatted JSON +``` + +## Frontend Validation + +Once backend has fresh data: + +```javascript +// In News.tsx, you should see: +// 1. Recent timestamps in console logs +// 2. Current dispatch content +// 3. Proper sorting (newest first) +// 4. No "No dispatches" fallback message +``` + +## Debugging Tips + +### Check Database Contents + +```bash +# Open SQLite CLI +sqlite3 helldivers2.db + +# See table info +.tables +.schema dispatches + +# See latest dispatches sorted by published date +SELECT * FROM dispatches ORDER BY json_extract(data, '$.published') DESC LIMIT 5; + +# Count total dispatches +SELECT COUNT(*) FROM dispatches; +``` + +### Monitor Collector Logs + +```bash +# In your terminal running `make dev`, watch for: +# [INFO] Collector: Collecting all game data +# [INFO] Collector: Collected X dispatches +# [ERROR] Collector: Failed to fetch... +``` + +### Check API Response + +```bash +# Get raw API response with headers +curl -v http://localhost:5000/api/dispatches?limit=20 + +# Check timestamps in response +curl http://localhost:5000/api/dispatches | jq '.[0].published' +``` + +## Next Steps + +1. **Immediate**: Run manual refresh endpoint + ```bash + curl -X POST http://localhost:5000/api/dispatches/refresh + ``` + +2. **Verify**: Check database timestamps + ```bash + sqlite3 helldivers2.db "SELECT published FROM dispatches ORDER BY published DESC LIMIT 1;" + ``` + +3. **Monitor**: Watch collector logs for auto-updates + ```bash + # Keep terminal open during `make dev` + ``` + +4. **Frontend**: Refresh browser to see new data + +## Support + +If data still doesn't update: + +- Check `.env` file for correct API headers (SUPER_CLIENT_HEADER, SUPER_CONTACT_HEADER) +- Verify network connectivity: `ping api.helldivers2.dev` +- Check rate limits haven't been exceeded (wait 10 minutes before retry) +- Review Hell Divers 2 API status: https://api.helldivers2.dev + +--- + +**Remember**: The frontend and API logic are working correctly. This is purely a data freshness issue in the backend database. Once the scraper collects current data, everything will display properly. diff --git a/docs/README.md b/docs/README.md index 7423dbf..d209928 100644 --- a/docs/README.md +++ b/docs/README.md @@ -69,6 +69,15 @@ Advanced filtering options for mission data: - Planet Events: sort, location filtering, event type filtering - Designed for both MCP and website use cases +### [BACKEND_DATA_REFRESH.md](./BACKEND_DATA_REFRESH.md) - Backend Data & Collector +Backend data collection and troubleshooting: +- Background collector scheduling (every 5 minutes) +- Manual refresh endpoints for testing +- Database data freshness verification +- Troubleshooting stale or missing data +- Upstream API connectivity checking +- Perfect for when data seems outdated + ### [QUICKREF.md](./QUICKREF.md) - Quick Reference Quick lookup card with: - Common curl commands @@ -92,6 +101,8 @@ Quick lookup card with: | Configure settings | [CONFIGURATION.md](./CONFIGURATION.md) (if exists) | | Quick command reference | [QUICKREF.md](./QUICKREF.md) | | Debug integration issues | [ARCHITECTURE.md](./ARCHITECTURE.md) → [IMPLEMENTATION.md](./IMPLEMENTATION.md) | +| Debug stale/missing data | [BACKEND_DATA_REFRESH.md](./BACKEND_DATA_REFRESH.md) | +| Verify collector status | [BACKEND_DATA_REFRESH.md](./BACKEND_DATA_REFRESH.md) | ### By Component @@ -100,10 +111,11 @@ Quick lookup card with: | **API Endpoints** (FastAPI) | [ENDPOINTS.md](./ENDPOINTS.md) + [API.md](./API.md) | | **Data Scraper** (HTTP client) | [IMPLEMENTATION.md](./IMPLEMENTATION.md) (rate limiting section) | | **Database** (SQLite) | [IMPLEMENTATION.md](./IMPLEMENTATION.md) (database schema & CRUD) | -| **Background Collector** | [ARCHITECTURE.md](./ARCHITECTURE.md) + [IMPLEMENTATION.md](./IMPLEMENTATION.md) | +| **Background Collector** | [ARCHITECTURE.md](./ARCHITECTURE.md) + [IMPLEMENTATION.md](./IMPLEMENTATION.md) + [BACKEND_DATA_REFRESH.md](./BACKEND_DATA_REFRESH.md) | | **Rate Limiting** | [IMPLEMENTATION.md](./IMPLEMENTATION.md) | | **Error Handling** | [IMPLEMENTATION.md](./IMPLEMENTATION.md) + [ARCHITECTURE.md](./ARCHITECTURE.md) | | **Caching & Fallback** | [IMPLEMENTATION.md](./IMPLEMENTATION.md) | +| **Data Freshness** | [BACKEND_DATA_REFRESH.md](./BACKEND_DATA_REFRESH.md) | ### By Audience diff --git a/src/database.py b/src/database.py index 5472666..8df77df 100644 --- a/src/database.py +++ b/src/database.py @@ -341,12 +341,16 @@ def get_latest_assignments(self, limit: int = 10) -> List[Dict]: return [] def get_latest_dispatches(self, limit: int = 10) -> List[Dict]: - """Get latest dispatches""" + """Get latest dispatches sorted by published date""" try: with sqlite3.connect(self.db_path) as conn: cursor = conn.cursor() + # Create index on published date for better performance if not exists cursor.execute( - "SELECT data FROM dispatches ORDER BY timestamp DESC LIMIT ?", + "CREATE INDEX IF NOT EXISTS idx_dispatches_published ON dispatches(json_extract(data, '$.published'))" + ) + cursor.execute( + "SELECT data FROM dispatches ORDER BY json_extract(data, '$.published') DESC LIMIT ?", (limit,), ) results = cursor.fetchall()