Retry HTTP 503 responses

[winklemad]

No linked issue — found by reading the code.

The bug

retry_if_specific_error (esm/sdk/retry.py) lists the retryable HTTP status codes as:

exception.error_code in {
    429,
    500,
    502,
    504,
    500,   # <- 500 listed twice; 503 is missing
}

500 appears twice and 503 is absent, so the set is effectively {429, 500, 502, 504}. A transient HTTP 503 Service Unavailable is therefore never retried — even though its sibling upstream errors 502 Bad Gateway and 504 Gateway Timeout are. 503 is the canonical "try again later" status (load-balancer draining, deploy rollover, server overload, and it often carries Retry-After), so a request that hits one is returned to the caller as an ESMProteinError on the very first attempt instead of being retried with backoff.

base_forge_client sets error_code from the response status, so a real 503 arrives here as exactly 503:

from esm.sdk.api import ESMProteinError
from esm.sdk.retry import retry_if_specific_error

retry_if_specific_error(ESMProteinError(error_code=502, error_msg="")) # True
retry_if_specific_error(ESMProteinError(error_code=503, error_msg="")) # False  <- bug
retry_if_specific_error(ESMProteinError(error_code=504, error_msg="")) # True

The duplicate 500 is the smoking gun — a status code isn't listed twice on purpose; one of those slots was meant to be 503.

Fix

Replace the duplicate 500 with 503, making the set {429, 500, 502, 503, 504}.

Testing

Added esm/sdk/retry_test.py asserting that the transient statuses 429/500/502/503/504 are retried, and that a non-transient client error (404) and a non-ESMProteinError exception are not. The 503 case fails before this change and passes after; ruff check / ruff format are clean.