Repository navigation
feat(api): opt-in retries that follow the shared retry policy, off by default - #43
Conversation
|
Same retry policy across all clients (429/502/503/504 and pre-response connection errors; 2 retries by default; full jitter; Retry-After capped at 30 s; 90 s total budget; timeouts never retried):
|
jonathan-zenrows
left a comment
There was a problem hiding this comment.
The retry logic looks right to me, but the branch is stale.
1. It breaks once rebased onto main
Main has #33, which removed the AutoParse fallback. GitHub says the merge is clean, but on the merged tree:
git checkout feat/api-retry-jitter && git merge origin/main
go test -race ./...
# FAIL TestExtractFallbackSharesTheRetryBudget (retry_internal_test.go:226) got 402 after 2 calls
# FAIL TestExtractRetriesThenFallsBackOnAuth010 (retry_test.go:413)
CI only ran on the branch head, so it stays green. After the rebase, delete those two tests and the fallback wording in client.go:171-174, README.md:395 and retry.go:44-45. callBudget/withCallBudget then have nothing to share, so the budget start can live in retryCall.
2. A bad option now fails every call instead of failing at startup
WithRetryWaitTime(0) was a reasonable "don't wait" setting before this PR. After upgrading, every request returns InvalidOptionError, with nothing at compile time and nothing in NewClient. Either clamp 0 up to the 250ms floor, or fail in NewClient.
3. retryAfter falls back to 0 (retry.go:123-128)
In resty, 0 means "use your own backoff". With SetRetryWaitTime(0) + SetRetryMaxWaitTime(math.MaxInt64) that works out to about 2^62 ns. It can't happen today, because shouldRetry always sets nextWait >= 250ms. But the first new retry condition someone adds would hit it. Return minRetryWait instead.
4. Release note
As feat(api): this is 1.4.0 with no note. ! doesn't help either, because it would force a /v2 module path. So the default flip, plus WithRetryWaitTime going from 5s to 1s, needs to be written into the release by hand.
Question: the README says to put Idempotency-Key in CustomHeaders, which sets custom_headers=true and forwards it to the target. Does the API dedupe on that header, or does it only reach the target site? If the API doesn't dedupe, retrying a POST isn't actually safe.
… default Retries stay off by default: a client that sets no retry count sends each request once, as in v1.3.0, and its other retry options are not checked. With WithMaxRetryCount above 0, only a GET is retried: on a 429, 500, 502, 503 or 504, a 422 other than no_data_extracted, or a failed dial. The wait is the API's Retry-After (250ms to 30s) or a full-jitter backoff, and no retry starts after WithRetryBudget (90s). Invalid retry options and a negative count return InvalidOptionError.
3a9f322 to
7681c85
Compare
Merge after #52, which adds the changelog section this PR's changelog lines use.
The API client follows the retry policy shared by the ZenRows clients when you turn retries on with
WithMaxRetryCountabove 0. Retries stay off by default: a client that sets no retry count sends each request once and sees no change, whatever other retry options it passes. A newWithRetryBudgetoption joins the existing three.When retries are on
GET(Get,Fetch,Extract,ScrapewithGET) is retried, and only when its body is not anio.Reader.429,500,502,503or504, a422other thanno_data_extracted, and a dial that failed (refused, connect timeout, unreachable network, name not resolved).ReturnOriginalStatusis set or a status listed inAllowedStatusCodes, andPrepareDomain/PreparedDomainStatus.Retry-Afterin whole seconds, clamped to 250ms-30s. Otherwise uniform(250ms, max(250ms, min(RetryMaxWaitTime, RetryWaitTime * 2^n))), with n = 0 for the first retry.WithRetryBudget(90s by default) from the first attempt. Canceling the context ends the wait.If you already set a retry count
502,503and504are now retried.POSTorPUTis never retried, on any status or error.GETwith anio.Readerbody is no longer retried.ReturnOriginalStatus,AllowedStatusCodes) is no longer retried.WithRetryWaitTimeis the backoff base (default 1s), not a minimum wait (was 5s). Every wait is at least 250ms, andWithRetryWaitTime(0)means 250ms.Retry-Afterfrom the API sets the wait, up to 30s, even aboveWithRetryMaxWaitTime.WithRetryWaitTime,WithRetryMaxWaitTimeorWithRetryBudgetmakes every scrape call returnInvalidOptionErrorwhen a count is set.WithMaxRetryCount(-1)returnsInvalidOptionErrorinstead of panicking with a nil pointer.Tests
Retry-Afterand jitter vectors, and for which errors count as a failed dial.200. A compile-time check keepsClientcomparable, and apidiff against v1.3.0 andmainshows only additions.go mod tidy,gofmt -s,go vet,go test -raceandgolangci-lint --new-from-rev=origin/mainpass in all three modules.BEGIN_COMMIT_OVERRIDE
feat(api): opt-in retries following the shared retry policy; still off by default
behaviour(api): with a retry count set, 502/503/504 are retried; POST, PUT, io.Reader bodies, target-site statuses and errors after sending are not; waits use a 1s jittered backoff base, Retry-After up to 30s and a 90s budget; invalid retry options return InvalidOptionError
END_COMMIT_OVERRIDE