Repository navigation
feat(scheduler): bounded retry for transient Threads API errors + carousel failure enrichment (#257) - #257
Merged
Conversation
…ousel failure enrichment (#257) ## What - MIGRATION 018: schedules.attempt_count (int, default 0) + schedules.next_due_at (nullable RFC3339 UTC eligibility gate). - SCHEDULER: on publish failure, classify the error via TitenError::is_transient_message. Transient + non-bundle + attempts < 2 defer the schedule 10 minutes (pending + [retry N] error marker); otherwise fail permanently with attempt_count persisted. - CLASSIFIER: transient = OAuthException #1 (Meta catch-all, killed carousel child creation on 2026-09-08 and 2026-09-10 while TEXT/IMAGE kept publishing), OAuthException #24 (observed transient 2026-08-25), HTTP 429/5xx. Permanent = #100, #190, validation errors. - GET_DUE_SCHEDULES: next_due_at overrides scheduled_at as the eligibility gate when set — otherwise a retry would fire on the very next tick and the backoff would never hold. NULL keeps legacy behavior. - PUBLISHER: carousel child-container failures now surface WHICH child failed (idx/total), the offending URL, and orphaned child IDs — every prior failure looked identical and required guesswork. - Bundle members are excluded from retry: a chain that breaks mid-flight fails the whole chain (existing semantics preserved). ## Why Production audit of 2026-09-08 + 2026-09-10 carousel failures (OAuthException #1 at child-container creation) showed Meta-side transient errors permanently kill a scheduled slot with zero recovery — the identical content republishes fine on a later attempt. Two back-to-back carousel slots died silently; the same rows failed on two different Titen builds, ruling out a code regression. ## Testing - 7 new unit tests for is_transient_message (transient + permanent cases) - 4 new integration tests (store_retry.rs): deferral sets a future gate and hides the row from get_due_schedules; gate expiry re-exposes it; legacy rows (NULL gate) keep scheduled_at semantics; final failure persists attempt_count and clears the gate. - cargo test --workspace green; clippy -D warnings clean; fmt clean.
Open
2 tasks
🔍 Cora AI Code ReviewReview powered by cora-code · BYOK · MIT |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
schedules.attempt_count(int, default 0) +schedules.next_due_at(nullable RFC3339 UTC eligibility gate).TitenError::is_transient_message. Transient + non-bundle + attempts < 2 → defer 10 minutes (pending+[retry N]error marker); otherwise fail permanently with attempt count persisted.next_due_atoverridesscheduled_atwhen set — otherwise a retry would fire on the very next tick and the backoff would never hold. NULL keeps legacy behavior.Why
Production audit of the 2026-09-08 + 2026-09-10 carousel failures (OAuthException #1 at child-container creation) showed Meta-side transient errors permanently kill a scheduled slot with zero recovery — the identical content republishes fine on a later attempt. Two carousel slots died silently; both failed on different Titen builds (v0.9.1 process and v0.9.2), ruling out a code regression.
Testing
is_transient_message(transient + permanent cases).store_retry.rs): deferral sets a future gate and hides the row fromget_due_schedules; gate expiry re-exposes it; legacy rows (NULL gate) keepscheduled_atsemantics; final failure persistsattempt_countand clears the gate.cargo test --workspacegreen;cargo clippy --workspace --all-targets -- -D warningsclean;cargo fmtclean.