[surrealwolf]

Problem

The /api/dispatches endpoint was returning dispatches sorted by database insertion timestamp instead of by the dispatch's actual published date. This caused old March 2024 dispatches to appear first even though current October 2025 dispatches existed in the database.

Root Cause

The get_latest_dispatches() method in src/database.py was using:

ORDER BY timestamp DESC

This sorts by when the row was inserted into the database, not when the dispatch was actually published. Since the collector updates all 635 dispatches together, they all have the same recent database timestamp despite containing data from different historical dates.

Solution

Changed the query to sort by the dispatch's published date from the JSON data:

ORDER BY json_extract(data, '$.published') DESC

Now dispatches are correctly sorted by their publication date, with newest first.

Changes

• src/database.py: Updated get_latest_dispatches() to sort by json_extract(data, '$.published') instead of timestamp
• docs/BACKEND_DATA_REFRESH.md: Added comprehensive troubleshooting guide for backend data collection and freshness
• docs/README.md: Added references to the new backend data refresh documentation

Testing

✅ Verified the SQL query returns October 2025 dispatches first:

2025-10-21T22:26:51Z|3458  (most recent)
2025-10-21T12:36:46Z|3457
2025-10-20T16:51:32Z|3455
...

✅ Collector is running every 5 minutes and collecting 635 current dispatches
✅ Upstream Hell Divers 2 API is online and providing fresh data

Impact

• Fixes dispatch ordering issue reported in frontend integration
• Improves data freshness visibility for API users
• No breaking changes to API contract
• Backward compatible with existing query parameters and sorting logic

[Copilot]

Pull Request Overview

Fixes the dispatch sorting issue where old March 2024 dispatches appeared first instead of current October 2025 dispatches. The root cause was sorting by database insertion timestamp rather than the dispatch's actual published date. Since all 635 dispatches are updated together by the collector, they shared the same insertion timestamp despite different publication dates.

Key Changes:

• Modified dispatch query to sort by JSON-extracted published field instead of insertion timestamp
• Added comprehensive troubleshooting documentation for backend data collection issues

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
src/database.py	Updated get_latest_dispatches() to sort by json_extract(data, '$.published') instead of timestamp
docs/README.md	Added navigation links to new BACKEND_DATA_REFRESH.md documentation
docs/BACKEND_DATA_REFRESH.md	New troubleshooting guide for backend data collection, freshness verification, and collector status