Add retry logic for transient token request failures - #10
Open
c1-dev-bot[bot] wants to merge 1 commit into
Open
Conversation
…failures The dpop_oauth2 token source previously had no retry mechanism for transient errors (timeouts, connection resets, server 5xx errors). During long-running connector syncs (e.g., 7-10 hour Azure DevOps syncs), transient failures on the /auth/v1/token endpoint would cause immediate sync failure with no recovery. This change adds: - Configurable per-attempt timeout via WithTokenTimeout() (default 30s) - Configurable retry count via WithMaxRetries() (default 3) - Exponential backoff between retries (2s, 4s, 8s, ...) - Transient error classification (timeouts, connection errors, HTTP 5xx) - Non-retryable errors (auth errors, invalid tokens) still fail immediately The defaults maintain backward compatibility: existing callers get automatic retries without code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
dpop_oauth2token source previously had no retry mechanism for transient errors. When the token endpoint (/auth/v1/token) experienced timeouts, connection resets, or returned HTTP 5xx errors, the token request would fail immediately with no recovery attempt. This is particularly problematic during long-running connector syncs (7-10+ hours) where transient network failures become increasingly likely.Changes
Token()now retries on transient errors with exponential backoff (2s, 4s, 8s, ...)WithTokenTimeout(d)option to set per-attempt timeout (default 30s, unchanged)WithMaxRetries(n)option to set max retry attempts (default 3, set 0 to disable)Backward Compatibility
Existing callers automatically get retry behavior with the defaults (3 retries, 30s per-attempt timeout). No code changes needed to benefit from this fix.
Test plan
TestTokenSource_RetryOnServerError- verifies retry succeeds after transient 504 errorsTestTokenSource_RetryExhausted- verifies proper error after all retries failTestTokenSource_NoRetryOnAuthError- verifies auth errors are not retriedTestTokenSource_NoRetryWhenDisabled- verifiesWithMaxRetries(0)disables retriesTestIsTransientError- verifies transient error classificationTestWithTokenTimeout/TestDefaultRetrySettings- verifies option plumbingAutomated PR Notice
This PR was automatically created by c1-dev-bot as a potential implementation.
This code requires: