Retry transient failures in Xylem Data Lake adapter#221
Merged
Conversation
Vendor-side instability this past week killed standard DAG runs in two
places that previously had no retry:
- 5xx from /api/v1/sqllab/execute/ during chunk queries (fatal at
raise_for_status)
- 401 from /api/v1/me/ during initial Superset login (fatal in
_create_superset_session)
Adds retry-with-backoff (10s/20s/40s, 3 attempts) at both points, plus
the same retry path for ConnectionError and Timeout exceptions on the
query POST. Re-auths triggered mid-query also flow through the new
auth-retry wrapper.
dmarulli
requested review from
christophertull,
jin-current,
mdowell12 and
pat-current
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds retry-with-backoff at the two points in the Xylem Data Lake adapter that have killed standard DAG runs this past week. Same pattern (10s/20s/40s, max 3 attempts) at all three points.
Failures observed (Crescent, May 3–7)
500 INTERNAL SERVER ERRORfrom/api/v1/sqllab/execute/mid-run_queryatraise_for_status()401from/api/v1/me/after successful Keycloak handshake_create_superset_sessionat the unexpected-shape checkIn both 5xx cases the in-query session-expired path successfully re-authenticated just before the run died — so the existing re-auth mechanism is fine. The gap is that 5xx and initial-auth failures had no retry.
Changes
_query— newserver_failurescounter, triggered when500 ≤ status < 600, backs off and retries before falling through toraise_for_status().ConnectionError/Timeoutretry in_query— extends the existingtry/except(which only caughtTooManyRedirects) to route transient network failures through the same backoff path._create_superset_session_with_retrywrapper — used for both the initial auth call in_extractand for re-auths from_reauth. CatchesRuntimeError(e.g., the unexpected-shape 401) andRequestException, retries with backoff.Test plan