Repository navigation
Fix the four critical findings from the 15 November performance review - #311
Merged
Merged
Conversation
A client could send all but the last byte of a body and stop. readBody kept every chunk until the body ended, and the server set no timeouts, so Node's five-minute requestTimeout applied: at the write limit of 60 requests a minute, one address could hold about 300 MB of the oracle, and three could pass the container's 768 MiB. - readBody drops a body still arriving after 10 s, with its connection, and the route answers 408. - POST /attest, the public write route, takes at most 32 KB. The largest real body, a payer outcome with a 4,096-byte ERC-1271 signature in hex, is about 10 KB. The loopback routes keep 1 MB. - The server allows 10 s for headers and 15 s for a whole request, as the signer already does. Neither bounds an answer, so a long /evidence is not cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… flight, and keep /score from starting long reads Payer standings were read at the latest block and the cache was emptied at every epoch, so every payer was read again every hour. Reads in flight were not shared, and each GET /score read every payer it found missing when it started: measured, 30 requests 300 ms apart for an agent with 300 payers made 4,650 reads in 45 s, against the 6.7 a second the epoch keeps to. /health's epochRunning let the burst be timed to land on the epoch, whose reads then failed and whose agents were skipped. - Standings are read at the snapshot block, the start of the UTC day containing the score's time, as the external factor's reviewers already are, and kept until the snapshot moves (today's and yesterday's). A checker auditing a proposal made the same day reads what its primary read. This changes what a standing is, so both operators deploy it together. - One read per payer at a time, shared by the epoch and every request, and the cache is checked as each chunk is made up. - GET /score reads standings for itself only between epochs, one chunk a request, and no faster than the epoch reads (two chunks of burst, refilled at its pace). Past that it answers 503 standings_pending with Retry-After, and the epoch reads the rest. The two pacing tests now run through an epoch, since /score no longer reads 25 payers in one request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The payment-chain scan confirmed each credited payment with two round trips, one after the other: the recipient's code at that block, then the block's time. A step of 1,000 reads took about 150 s at Base Sepolia's 145 ms, so payments in more blocks an hour than the 10-minute budget could confirm (roughly 1,000 to 3,000 of Base Sepolia's 18,000) put the scan behind, and a chain behind the epoch's cut-off holds every agent (ADR 0009). Paying your own agent the 0.01 USDC minimum gets the money back, so on Base Sepolia that cost only free gas. Each block's time and its recipients' code now go out together, eight blocks at a time, which ethers sends as one JSON-RPC batch, paced as the log reads are. The endpoint the operators use (publicnode) answered batches of 100 such reads in 140 to 330 ms on 15 Nov 2026; eight blocks ask about sixteen. A block's time is also read when all its payments turn out to be to a contract, since it rides in the same batch. What is credited, and when, is unchanged. This raises the rate needed to trip the hold several times over; the rule that a lagging chain holds everyone stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scored chain's payment scan read each credited block's time one round trip at a time, with no pace. Past Arc's bucket of about 60 requests the reads crawled on backoff, so payments in enough blocks could keep the scan behind, and a scan far enough behind holds every agent, the same way a payment chain's does. The reads now go ten blocks at a time, half of the 20 calls Arc answers in one batch, paced as the scan's log reads are. What is credited, and when, is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and hold signatures to 1,024 bytes load() read the state file into one string and parsed it while the string was alive, so a load peaked at about 2.2 times the file. V8's heap in the 768 MiB container is 432 MB (measured on the primary), and the caps had been sized against the 768. A state at the caps where every payment carries an outcome (182 MB) ended the process inside JSON.parse, and a payer could get there with far fewer payments by padding its outcomes with 4,096-byte ERC-1271 signatures: about 24,000 of them broke a restart. The checker imports outcomes, so both operators failed, an out-of-memory abort is not an exception, so the .bak was never tried, and the restart policy started each into the same failure. - The file is read as bytes, outside V8's heap, and parsed a piece of at most 1 MB at a time (parseJsonBuffer): a container larger than that is walked member by member, and runs of small members are parsed together. Same result as JSON.parse, refuses what it refuses, and the format is unchanged, so an older build reads what this one writes. - Each agent's ledger is written in slices of 256, the same bytes as before. It was one string: 11 MB at the caps, 151 MB with padded outcomes. - A payer outcome's signature is held to 1,024 bytes. The longest held on the primary on 15 Nov 2026 was 640, a smart account's. Both operators change this together, or one refuses imports the other accepted. Measured under a 432 MB heap, synthetic state at the caps: today's mix (157 MB) loads with 207 MB of heap instead of 346 MB, in 2.0 s instead of 1.2 s; every payment with an outcome (182 MB) loads with 220 MB and 458 MB RSS, where it ran out of heap before. Not changed: the caps still count payments, not bytes, so a ledger filled to 150,000 payments that all carry 1,024-byte signatures is still too large for the heap. That takes about 150,000 payments from an ERC-1271 payer and needs caps that weigh bytes, which changes what both operators drop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d count 1514 oracle tests - scoreStandingsPending is declared and exported, as every metric the source increments must be. - The answer-size tests' fake endpoint answers a JSON-RPC batch call by call. The payment-chain scan now asks for its blocks' times and code together, and ethers sends those as one batch; the real endpoint answers batches (measured). - README and whitepaper: 1514 oracle tests, from 1500. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four critical findings from the 15 November performance and security review of main 93a77d9. Each is closed by a test that fails on the old code (run against it before committing).
What was wrong, and what changes
1. A flood of small payments held every agent's score (
payment-chains.js,index.js)2. Payer standings were read again every epoch, and GET /score multiplied the reads (
index.js,chain.js)epochRunninglet the burst be timed to land on the epoch, whose reads then failed and whose agents were skipped.standings_pendingwith Retry-After, and the epoch reads the rest. Nothing in the repo calls /score.3. A stalled POST /attest body held up to 1 MB for 5 minutes (
http-helpers.js,index.js)readBodykept every chunk until the body ended, and the server set no timeouts. At 60 requests a minute, one address could hold about 300 MB; three could pass the container's 768 MiB.4. A restart could not load a state near the caps (
state-serialize.js,store.js,payer-outcome.js)parseJsonBuffer). The result is the same as JSON.parse, it refuses what JSON.parse refuses, and the format is unchanged, so an older build reads what this one writes.Measured
Deploy
Both operators together. Standings at the snapshot block change what a standing is, and a checker on the old build would read them at the latest block. A checker that holds the 1,024-byte signature limit refuses imports of longer ones that a primary on the old build accepted.
The state format is unchanged, so rolling back is safe.
No contract changes. The keeper, publisher, signer, indexer and watcher are untouched.
Separately, a live Caddy change on archipelago-standby, to stop oversize bodies at the edge. It goes inside the oracle.sigvara.xyz block only, since this Caddy also serves other sites:
Not changed
Tests
1,514 oracle tests pass locally on Windows, Node 22 (the first full run caught an undeclared metric and a test RPC that could not answer a batch, both fixed in the last commit). The new ones:
🤖 Generated with Claude Code