Skip to content

feat(api): opt-in retries that follow the shared retry policy, off by default - #43

Merged
gonzalo-zenrows merged 1 commit into
mainfrom
feat/api-retry-jitter
Nov 6, 2026
Merged

gonzalo-zenrows merged 1 commit into
mainfrom
feat/api-retry-jitter

Conversation

@gonzalo-zenrows

@gonzalo-zenrows gonzalo-zenrows commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Merge after #52, which adds the changelog section this PR's changelog lines use.

The API client follows the retry policy shared by the ZenRows clients when you turn retries on with WithMaxRetryCount above 0. Retries stay off by default: a client that sets no retry count sends each request once and sees no change, whatever other retry options it passes. A new WithRetryBudget option joins the existing three.

When retries are on

  1. Only a GET (Get, Fetch, Extract, Scrape with GET) is retried, and only when its body is not an io.Reader.
  2. Retried: a 429, 500, 502, 503 or 504, a 422 other than no_data_extracted, and a dial that failed (refused, connect timeout, unreachable network, name not resolved).
  3. Never retried: a reset or EOF after sending, a read timeout, a canceled context, any status when ReturnOriginalStatus is set or a status listed in AllowedStatusCodes, and PrepareDomain / PreparedDomainStatus.
  4. Wait: the API's Retry-After in whole seconds, clamped to 250ms-30s. Otherwise uniform(250ms, max(250ms, min(RetryMaxWaitTime, RetryWaitTime * 2^n))), with n = 0 for the first retry.
  5. No retry starts after WithRetryBudget (90s by default) from the first attempt. Canceling the context ends the wait.

If you already set a retry count

  1. 502, 503 and 504 are now retried.
  2. A POST or PUT is never retried, on any status or error.
  3. Only a failed dial is retried. A reset or EOF after sending, a read timeout, a TLS error or a proxy error is no longer retried.
  4. A GET with an io.Reader body is no longer retried.
  5. A status from the target site (ReturnOriginalStatus, AllowedStatusCodes) is no longer retried.
  6. WithRetryWaitTime is the backoff base (default 1s), not a minimum wait (was 5s). Every wait is at least 250ms, and WithRetryWaitTime(0) means 250ms.
  7. A Retry-After from the API sets the wait, up to 30s, even above WithRetryMaxWaitTime.
  8. No retry starts after 90s.
  9. An invalid WithRetryWaitTime, WithRetryMaxWaitTime or WithRetryBudget makes every scrape call return InvalidOptionError when a count is set.
  10. WithMaxRetryCount(-1) returns InvalidOptionError instead of panicking with a nil pointer.

Tests

  1. One test per rule against a local server, plus table tests for the shared Retry-After and jitter vectors, and for which errors count as a failed dial.
  2. A client with no retry count and any other retry option sends once and gets its 200. A compile-time check keeps Client comparable, and apidiff against v1.3.0 and main shows only additions.
  3. go mod tidy, gofmt -s, go vet, go test -race and golangci-lint --new-from-rev=origin/main pass in all three modules.

BEGIN_COMMIT_OVERRIDE
feat(api): opt-in retries following the shared retry policy; still off by default

behaviour(api): with a retry count set, 502/503/504 are retried; POST, PUT, io.Reader bodies, target-site statuses and errors after sending are not; waits use a 1s jittered backoff base, Retry-After up to 30s and a 90s budget; invalid retry options return InvalidOptionError
END_COMMIT_OVERRIDE

@gonzalo-zenrows

Copy link
Copy Markdown
Contributor Author

@jonathan-zenrows jonathan-zenrows left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retry logic looks right to me, but the branch is stale.

1. It breaks once rebased onto main

Main has #33, which removed the AutoParse fallback. GitHub says the merge is clean, but on the merged tree:

git checkout feat/api-retry-jitter && git merge origin/main
go test -race ./...
# FAIL TestExtractFallbackSharesTheRetryBudget  (retry_internal_test.go:226)  got 402 after 2 calls
# FAIL TestExtractRetriesThenFallsBackOnAuth010 (retry_test.go:413)

CI only ran on the branch head, so it stays green. After the rebase, delete those two tests and the fallback wording in client.go:171-174, README.md:395 and retry.go:44-45. callBudget/withCallBudget then have nothing to share, so the budget start can live in retryCall.

2. A bad option now fails every call instead of failing at startup

WithRetryWaitTime(0) was a reasonable "don't wait" setting before this PR. After upgrading, every request returns InvalidOptionError, with nothing at compile time and nothing in NewClient. Either clamp 0 up to the 250ms floor, or fail in NewClient.

3. retryAfter falls back to 0 (retry.go:123-128)

In resty, 0 means "use your own backoff". With SetRetryWaitTime(0) + SetRetryMaxWaitTime(math.MaxInt64) that works out to about 2^62 ns. It can't happen today, because shouldRetry always sets nextWait >= 250ms. But the first new retry condition someone adds would hit it. Return minRetryWait instead.

4. Release note

As feat(api): this is 1.4.0 with no note. ! doesn't help either, because it would force a /v2 module path. So the default flip, plus WithRetryWaitTime going from 5s to 1s, needs to be written into the release by hand.

Question: the README says to put Idempotency-Key in CustomHeaders, which sets custom_headers=true and forwards it to the target. Does the API dedupe on that header, or does it only reach the target site? If the API doesn't dedupe, retrying a POST isn't actually safe.

@gonzalo-zenrows gonzalo-zenrows changed the title feat(api): retry transient failures by default with full-jitter backoff feat(api): opt-in retries with the shared retry policy, off by default Nov 2, 2026
… default

Retries stay off by default: a client that sets no retry count sends each
request once, as in v1.3.0, and its other retry options are not checked.

With WithMaxRetryCount above 0, only a GET is retried: on a 429, 500, 502,
503 or 504, a 422 other than no_data_extracted, or a failed dial. The wait
is the API's Retry-After (250ms to 30s) or a full-jitter backoff, and no
retry starts after WithRetryBudget (90s). Invalid retry options and a
negative count return InvalidOptionError.
@gonzalo-zenrows gonzalo-zenrows changed the title feat(api): opt-in retries with the shared retry policy, off by default feat(api): opt-in retries that follow the shared retry policy, off by default Nov 3, 2026
@gonzalo-zenrows
gonzalo-zenrows merged commit 1a54b91 into main Nov 6, 2026
5 checks passed
@gonzalo-zenrows
gonzalo-zenrows deleted the feat/api-retry-jitter branch November 6, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants