Skip to content

Try a node that refuses for its rate limit for longer, within a bound - #329

Merged
RunTimeAdmin merged 1 commit into
mainfrom
fix/rate-limit-backoff
Nov 16, 2026
Merged

RunTimeAdmin merged 1 commit into
mainfrom
fix/rate-limit-backoff

Conversation

@RunTimeAdmin

Copy link
Copy Markdown
Owner

#318 (finding 7) stopped ethers retrying an HTTP 429 inside each read, so a throttling node could no longer hold /score and /attest for minutes. Retries were left to readWithBackoff: four tries over 3.5 s. That is too short for the checker's node, rpc.quicknode.testnet.arc.io, which answers a burst with HTTP 429 and -32005 rate limit exceeded. Since the deploy the checker has left 2 to 4 agents unread each hourly epoch, and 10 to 34 at a restart: 52 of its retries and 9 of its agent errors were that refusal. The primary's node answers its rate limit inside a 200, which was already retried, and is unaffected.

What changes (read-retry.js, chain.js)

  • A refusal for the rate limit is tried up to 7 times. The wait doubles from 500 ms to at most 8 s, about 24 s at most in all. Each wait is between half and all of its step, so reads refused together do not come back together.
  • Every other failure keeps its four tries and its waits (0.5, 1, 2 s).
  • Still bounded: a read cannot hang for minutes, which is what Fix the oracle's part of the moderate findings from the 15 November performance review #318 fixed.
  • isRateLimited moves from chain.js to read-retry.js unchanged, since read-retry.js cannot load chain.js. chain.js imports it and still exports it.
  • readWithBackoff takes an optional { pause, random }, so the schedule can be tested without waiting.

Tests

chain.test.js adds two tests, both failing on main:

  • a 429 refusal tried 7 times, with waits doubling to 8 s and each at least half its step;
  • a -32005 nested as ethers nests it counts as a rate limit, the schedule ends after 7 tries, and any other failure keeps 4.

keeper.test.js: the keeper's process tests load test-support/fast-retry.js, which cuts waits of 250 ms or more to 10 ms. A keeper whose node refuses every read now fails each run only after the whole schedule, and six such runs no longer fit the test's minute. The production schedule is unchanged.

1,545 oracle tests: the full suite ran once, with one failure, that keeper test, which then passed with the shim (56 keeper tests). The documented 1,544 was one too many, because the minor round's run counted a failed test file as a test.

Deploy

Both operators. The checker is the one that needs it. The services pick it up at their next deploy.

🤖 Generated with Claude Code

#318 stopped ethers retrying an HTTP 429 inside each read, leaving retries to readWithBackoff: four tries over
3.5 s. The checker's node answers a burst with HTTP 429 and -32005, and since the deploy the checker has left 2 to
4 agents unread each hourly epoch. A rate-limit refusal is now tried up to 7 times, the wait doubling to at most
8 s and jittered, about 24 s at most; every other failure keeps four tries. isRateLimited moves to read-retry.js,
unchanged, so readWithBackoff can ask it.

The keeper's process tests load a shim that cuts long waits, so a run that fails only after the whole schedule
does not stretch the test. Oracle tests: 1545 (the documented 1544 counted a failed test file as a test).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RunTimeAdmin
RunTimeAdmin merged commit caa83a1 into main Nov 16, 2026
7 checks passed
@RunTimeAdmin
RunTimeAdmin deleted the fix/rate-limit-backoff branch November 16, 2026 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant