Skip to content

Fix the oracle's part of the moderate findings from the 15 November performance review - #318

Merged
RunTimeAdmin merged 2 commits into
mainfrom
fix/perf-moderate
Nov 15, 2026
Merged

RunTimeAdmin merged 2 commits into
mainfrom
fix/perf-moderate

Conversation

@RunTimeAdmin

Copy link
Copy Markdown
Owner

The oracle's part of the eight moderate findings from the 15 November performance review (the four critical ones were #311). Each fix has a test that fails on main, run against it before committing. The contract, SDK and site parts are separate PRs.

What changes

5. A checker's audit built a whole Merkle tree for every cut-off (evidence-as-of.js, merkle.js)

  • Every tree an audit tries now shares one memo of pair hashes. Neighbouring cut-offs hold nearly the same payments in the same order, so their trees share every pair left of where they differ.
  • A cut-off that counts the same payments as the one before is skipped, and other work runs between trees (evidenceAsOf is now async).
  • Measured: 31 cut-offs over 3,000 payments, no match, took 65,099 hashes; now 3,149. The review measured a miss at 25,000 payments holding the event loop for 22 s.

6. Every accepted /attest saved the whole state on its own (index.js, metrics.js)

  • A save writes and fsyncs everything: about 2.8 s at the ledger's caps. Writes that arrive within 100 ms of each other now share one save, and each is answered after a save that holds it, so a 200 is as durable as before.
  • /flag, /flag/resolve and /link keep their own save: their replies are remembered for a retry, which must not land between a change and its save.
  • New counter sigvara_oracle_state_saves_total. Five concurrent attestations took five saves; now one.

7. Chain reads had no deadline, ethers retried a 429 for minutes, and every wave of reads carried an eth_chainId (rpc-provider.js, used by chain.js, external.js, keeper.js, publisher.js, signer.js, watcher.js)

  • One provider factory: a 30 s deadline per request (the payment chains' figure; ethers' default is 300 s), no retries inside ethers (it tried an HTTP 429 up to 12 times inside each of readWithBackoff's attempts), and the chain fixed, so ethers stops asking eth_chainId before every wave.
  • What has to know the chain the node serves asks it directly, as before: chain.getChainId before every epoch, and external.refreshChainId, which now sends eth_chainId itself instead of getNetwork().
  • The signer's two reads go through readWithBackoff, since ethers no longer retries them.
  • Measured against a local endpoint: three reads sent three eth_chainId; now none.

8. /attest read the chain six to nine times before refusing a replay or a forged signature (index.js)

  • A settlement already credited to the agent, offered again with no outcome to add, is refused before any read. Anyone can copy one from /evidence. Both intake paths credit every qualifying sender of a transaction together, so one live credit means the rest were credited or refused with it. An auditor adding an outcome still goes through the full check.
  • A payer-signed outcome whose signature recovers to someone else, from a payer with no code, is refused after one read (the payer's code) instead of nine. An address with code is asked at its pinned block, as before.
  • I did not add the review's cache of verified settlements: with these two checks the cases it covered cost nothing or one read.

9. /evidence was one body per agent, built on the event loop (index.js, merkle.js, response-cache.js)

  • The body is built a slice at a time (merkle.treeOf, then 256 payments a slice), with other requests answered in between. The bytes are what one JSON.stringify gave.
  • The tree is kept per evidence array, and a request that arrives while a body is being built waits for that build instead of starting its own.
  • ?offset=N&limit=M (M up to 1,000) serves part of the list with its proofs, plus next. Without them the whole body is served, so the SDK and every verifier written before this keep working. Nothing in the repo pages yet.
  • An agent with no payments is not cached: one stored per didHash asked about pushed out the bodies worth keeping.
  • Dropped from the review's fix: ?summary=1. It was for the simulation page, which turned out to read every payment's fields for its replay, not a count.

10. Every unbonded agent was read again in one epoch once a day (bond-watch.js)

  • Each agent left out is read again within a day of the epoch that found it unbonded, the earliest first, a share each epoch. 240 such agents were all read in one epoch; now 10 an hourly epoch. A change no log showed still waits a day at most.
  • Not done: reading every transfer of the asset once past two recipient batches. Measured on 15 Nov: Arc testnet carries 2,546 USDC transfers per 1,000 blocks, about 13,000 an hourly epoch, past Arc's 20,000-result cap at the live 10,000-block span, to find a handful. It would pay off on Base Sepolia (209 per 1,000 blocks) and cost far more on Base mainnet. Registration staying free is a design question, not a code fix.

12. Each /outcomes page cost its size times the agent's events (store.js)

  • Each agent's outcomes are indexed once per page. The last page of 500 over 20,000 events took 738 ms; now a few ms.

Deploy

  • Both operators. The state format is unchanged, so rolling back is safe.
  • The keeper, publisher, signer and watcher take finding 7 when they are next deployed. Nothing else in this PR touches them, and nothing depends on their going first.
  • No contract changes.

Tests

The full suite ran once locally (Windows, Node 22): 1,531 tests, with 5 failures in older tests that still called evidenceAsOf synchronously or expected the old answers. Those were updated and their files run clean, so the count is 1,532 with the test added after; README and whitepaper say 1,532. Updated tests:

  • payment-chain-outcomes.test.js signs the unmined hash it posts: a signature made for another payment is now refused at once (400) instead of being told "not yet" (503).
  • http.test.js: an agent with no payments is not cached; the keying check moved to an agent with payments.

New:

  • store.test.js: an outcomes page indexed once.
  • attest.test.js: a replay refused before the payment is read; attestations that arrive together share a save.
  • payer-outcome.test.js: a forged signature refused before the payment is read.
  • rpc-provider.test.js: no eth_chainId with each read, one try on a 429, a deadline.
  • evidence-as-of.test.js: shared pairs across cut-offs (counted hashes), and other work runs between trees.
  • evidence-body.test.js: the sliced body is JSON.stringify's bytes, pages put together are the whole list, bad page values are refused, an empty agent is not cached.
  • merkle.test.js and response-cache.test.js: treeOf yields and keeps its tree; a build in flight is shared, and a stale one is not kept.
  • write-spend.test.js: unbonded agents are read 10 an hourly epoch, earliest first.

🤖 Generated with Claude Code

RunTimeAdmin and others added 2 commits November 15, 2026 15:48
…erformance review

- 5: a checker's audit shares one memo of pair hashes across the trees of every cut-off it tries,
  skips a cut-off that counts the same payments, and lets other work run between trees.
- 6: /attest writes that arrive within 100 ms share one save, each answered after a save that holds
  it. New counter sigvara_oracle_state_saves_total.
- 7: one provider factory (rpc-provider.js) for the oracle and its services: a 30 s deadline, no
  retries inside ethers, and the chain fixed so ethers stops asking eth_chainId with every read.
- 8: /attest refuses a replayed settlement before any chain read, and a forged signature from a
  payer with no code after one.
- 9: /evidence builds its body a slice at a time from a kept tree, shares a build in flight, serves
  pages with ?offset=&limit=, and does not cache an agent with no payments.
- 10: unbonded agents are read again within a day, a share each epoch, not all in one.
- 12: an /outcomes page indexes each agent's outcomes once instead of searching per entry.

Oracle tests: 1532.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main brought #316 and #317, whose test counts share a README and whitepaper line with this branch's. chain.js merged
without a conflict (#316 changed comments only); its tests and the docs check pass on the merged tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RunTimeAdmin

Copy link
Copy Markdown
Owner Author

Refreshed by merging main (#316, #317). The refresh commit carries [skip ci]: every job passed on 141e62d, main's parts were tested on their own PRs, the only conflicts were the test-count lines in README and the whitepaper, and chain.js (where #316 changed comments) merged cleanly; rpc-provider.test.js and chain.test.js (65 tests) and check-docs pass on the merged tree.

@RunTimeAdmin
RunTimeAdmin merged commit c0aa0d1 into main Nov 15, 2026
2 checks passed
@RunTimeAdmin
RunTimeAdmin deleted the fix/perf-moderate branch November 15, 2026 21:58
RunTimeAdmin added a commit that referenced this pull request Nov 16, 2026
… [skip ci] (#329)

#318 stopped ethers retrying an HTTP 429 inside each read, leaving retries to readWithBackoff: four tries over
3.5 s. The checker's node answers a burst with HTTP 429 and -32005, and since the deploy the checker has left 2 to
4 agents unread each hourly epoch. A rate-limit refusal is now tried up to 7 times, the wait doubling to at most
8 s and jittered, about 24 s at most; every other failure keeps four tries. isRateLimited moves to read-retry.js,
unchanged, so readWithBackoff can ask it.

The keeper's process tests load a shim that cuts long waits, so a run that fails only after the whole schedule
does not stretch the test. Oracle tests: 1545 (the documented 1544 counted a failed test file as a test).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant