fix: stop treating CheckMedia 429s as permanent upload failures - #629
Merged
Conversation
Rate-limited extractions were being written to uploadError and never retried, so a single burst of requests could permanently strand a large chunk of the upload backlog. Uploads are now paced, 429/5xx/network errors are left pending for the next scheduled run instead of being poisoned, and a run stops early once CheckMedia signals it's rate limited rather than cascading through the rest of the batch. Also adds a one-off script to clear uploadError on extractions that were already poisoned by a 429 before this fix, and a manual workflow_dispatch job to run maintenance scripts like it against a chosen environment via the existing migrator image. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The manual job-runner endpoint only ever ran Payload's default queue, so there was no way to nudge queues like exportSync (used by AI_EXTRACTION_EXPORT_ROWS_SYNC_CRON_SCHEDULE) without waiting for their own autoRun tick. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Rate-limited extractions were being written to uploadError and never retried, so a single burst of requests could permanently strand a large chunk of the upload backlog. Uploads are now paced, 429/5xx/network errors are left pending for the next scheduled run instead of being poisoned, and a run stops early once CheckMedia signals it's rate limited rather than cascading through the rest of the batch.
Also adds a one-off script to clear uploadError on extractions that were already poisoned by a 429 before this fix, and a manual workflow_dispatch job to run maintenance scripts like it against a chosen environment via the existing migrator image.
Fixes # (issue)
Type of change
Screenshots
Checklist: