Skip to content

Fix the four critical findings from the 15 November performance review - #311

Merged
RunTimeAdmin merged 6 commits into
mainfrom
fix/perf-review-critical
Nov 15, 2026
Merged

RunTimeAdmin merged 6 commits into
mainfrom
fix/perf-review-critical

Conversation

@RunTimeAdmin

Copy link
Copy Markdown
Owner

The four critical findings from the 15 November performance and security review of main 93a77d9. Each is closed by a test that fails on the old code (run against it before committing).

What was wrong, and what changes

1. A flood of small payments held every agent's score (payment-chains.js, index.js)

  • The payment-chain scan confirmed each credited payment with two round trips, one after the other. A step of 1,000 reads took about 150 s at Base Sepolia's 145 ms. Once payments landed in more blocks an hour than the 10-minute budget could confirm, the scan fell behind, and a chain behind the epoch's cut-off holds every agent (ADR 0009).
  • Paying your own agent the minimum gets the money back, so on Base Sepolia this cost only free gas.
  • Now each block's time and its recipients' code go out together, eight blocks to a batch, paced as the log reads are.
  • The endpoint the operators use for Base Sepolia (publicnode) answered batches of 100 such reads in 140 to 330 ms on 15 Nov.
  • The Arc scan read its block times one at a time with no pace. They now go ten at a time.
  • What is credited, and when, is unchanged. The rule that a lagging chain holds everyone stays; this raises the payment rate needed to trip it several times over.

2. Payer standings were read again every epoch, and GET /score multiplied the reads (index.js, chain.js)

  • The cache was emptied at every epoch, so every payer was read again every hour.
  • Reads in flight were not shared, and each /score request read every payer it found missing when it started. Measured: 30 requests for an agent with 300 payers made 4,650 reads in 45 s, against the 6.7 a second the epoch keeps to. /health's epochRunning let the burst be timed to land on the epoch, whose reads then failed and whose agents were skipped.
  • Standings are now read at the snapshot block (the start of the UTC day, as the external factor's reviewers already are) and kept until the snapshot moves. A checker auditing a proposal made the same day reads what its primary read.
  • One read per payer at a time, shared by the epoch and every request.
  • GET /score reads standings for itself only between epochs, one chunk a request, and no faster than the epoch reads. Past that it answers 503 standings_pending with Retry-After, and the epoch reads the rest. Nothing in the repo calls /score.

3. A stalled POST /attest body held up to 1 MB for 5 minutes (http-helpers.js, index.js)

  • readBody kept every chunk until the body ended, and the server set no timeouts. At 60 requests a minute, one address could hold about 300 MB; three could pass the container's 768 MiB.
  • Now a body still arriving after 10 s is dropped with its connection (408).
  • /attest takes at most 32 KB; the loopback routes keep 1 MB.
  • The server allows 10 s for headers and 15 s for a request, as the signer already does. Neither bounds an answer.

4. A restart could not load a state near the caps (state-serialize.js, store.js, payer-outcome.js)

  • The load read the file into one string and parsed it while the string was alive, peaking at about 2.2 times the file, against V8's 432 MB heap in the container (measured on the primary).
  • A payer could reach that with about 24,000 outcomes padded with 4,096-byte signatures. The checker imports outcomes, so both operators failed, and the restart policy restarted each into the same failure.
  • The file is now read as bytes and parsed a piece of at most 1 MB at a time (parseJsonBuffer). The result is the same as JSON.parse, it refuses what JSON.parse refuses, and the format is unchanged, so an older build reads what this one writes.
  • Each agent's ledger is written in slices of 256, the same bytes as before.
  • Signatures are held to 1,024 bytes. The longest held on the primary was 640, a smart account's.

Measured

Before After
Standing reads, 30 /score requests on one agent each request read all its payers one read per payer
Reads a payer costs per day 24 1
Base Sepolia payment reads in flight at once 1 16 (8 blocks)
Arc scan block reads in flight at once 1 10
Memory a stalled /attest request can hold 1 MB for 300 s 32 KB for 10 s
Load at the caps, today's mix (157 MB file), 432 MB heap 346 MB heap, 1.2 s 207 MB, 2.0 s
Load at the caps, every payment with an outcome (182 MB) out of heap 220 MB heap, 458 MB RSS, 2.3 s

Deploy

  • Both operators together. Standings at the snapshot block change what a standing is, and a checker on the old build would read them at the latest block. A checker that holds the 1,024-byte signature limit refuses imports of longer ones that a primary on the old build accepted.

  • The state format is unchanged, so rolling back is safe.

  • No contract changes. The keeper, publisher, signer, indexer and watcher are untouched.

  • Separately, a live Caddy change on archipelago-standby, to stop oversize bodies at the edge. It goes inside the oracle.sigvara.xyz block only, since this Caddy also serves other sites:

    handle @attest {
    	request_body {
    		max_size 64KB
    	}
    	# existing directives unchanged
    }

Not changed

  • The caps count payments, not bytes. A ledger filled to 150,000 payments that all carry 1,024-byte signatures is still too large for the heap. That takes about 150,000 payments from an ERC-1271 payer, and fixing it needs caps that weigh bytes, which changes what both operators drop.
  • A lagging payment chain still holds every agent (ADR 0009). The fix raises the rate needed.

Tests

1,514 oracle tests pass locally on Windows, Node 22 (the first full run caught an undeclared metric and a test RPC that could not answer a batch, both fixed in the last commit). The new ones:

  • http-helpers and http: the body deadline, and the 32 KB /attest limit.
  • payer-reads: standings read once a snapshot; shared reads; /score reads at most a chunk. The two pacing tests now run through an epoch.
  • chain: standing reads carry the snapshot block.
  • payment-chains and scan-checkpoint: batched block reads.
  • state-serialize and payer-outcome: the parser against JSON.parse, malformed input, slices, a 39 MB state loading under an 80 MB heap, and the signature limit.

🤖 Generated with Claude Code

RunTimeAdmin and others added 6 commits November 15, 2026 10:22
A client could send all but the last byte of a body and stop. readBody kept every
chunk until the body ended, and the server set no timeouts, so Node's five-minute
requestTimeout applied: at the write limit of 60 requests a minute, one address could
hold about 300 MB of the oracle, and three could pass the container's 768 MiB.

- readBody drops a body still arriving after 10 s, with its connection, and the
  route answers 408.
- POST /attest, the public write route, takes at most 32 KB. The largest real body,
  a payer outcome with a 4,096-byte ERC-1271 signature in hex, is about 10 KB. The
  loopback routes keep 1 MB.
- The server allows 10 s for headers and 15 s for a whole request, as the signer
  already does. Neither bounds an answer, so a long /evidence is not cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… flight, and keep /score from starting long reads

Payer standings were read at the latest block and the cache was emptied at every
epoch, so every payer was read again every hour. Reads in flight were not shared,
and each GET /score read every payer it found missing when it started: measured,
30 requests 300 ms apart for an agent with 300 payers made 4,650 reads in 45 s,
against the 6.7 a second the epoch keeps to. /health's epochRunning let the burst
be timed to land on the epoch, whose reads then failed and whose agents were
skipped.

- Standings are read at the snapshot block, the start of the UTC day containing
  the score's time, as the external factor's reviewers already are, and kept until
  the snapshot moves (today's and yesterday's). A checker auditing a proposal made
  the same day reads what its primary read. This changes what a standing is, so
  both operators deploy it together.
- One read per payer at a time, shared by the epoch and every request, and the
  cache is checked as each chunk is made up.
- GET /score reads standings for itself only between epochs, one chunk a request,
  and no faster than the epoch reads (two chunks of burst, refilled at its pace).
  Past that it answers 503 standings_pending with Retry-After, and the epoch reads
  the rest.

The two pacing tests now run through an epoch, since /score no longer reads 25
payers in one request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The payment-chain scan confirmed each credited payment with two round trips, one
after the other: the recipient's code at that block, then the block's time. A step
of 1,000 reads took about 150 s at Base Sepolia's 145 ms, so payments in more
blocks an hour than the 10-minute budget could confirm (roughly 1,000 to 3,000 of
Base Sepolia's 18,000) put the scan behind, and a chain behind the epoch's cut-off
holds every agent (ADR 0009). Paying your own agent the 0.01 USDC minimum gets the
money back, so on Base Sepolia that cost only free gas.

Each block's time and its recipients' code now go out together, eight blocks at a
time, which ethers sends as one JSON-RPC batch, paced as the log reads are. The
endpoint the operators use (publicnode) answered batches of 100 such reads in 140
to 330 ms on 15 Nov 2026; eight blocks ask about sixteen. A block's time is also
read when all its payments turn out to be to a contract, since it rides in the
same batch. What is credited, and when, is unchanged.

This raises the rate needed to trip the hold several times over; the rule that a
lagging chain holds everyone stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scored chain's payment scan read each credited block's time one round trip at
a time, with no pace. Past Arc's bucket of about 60 requests the reads crawled on
backoff, so payments in enough blocks could keep the scan behind, and a scan far
enough behind holds every agent, the same way a payment chain's does.

The reads now go ten blocks at a time, half of the 20 calls Arc answers in one
batch, paced as the scan's log reads are. What is credited, and when, is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and hold signatures to 1,024 bytes

load() read the state file into one string and parsed it while the string was
alive, so a load peaked at about 2.2 times the file. V8's heap in the 768 MiB
container is 432 MB (measured on the primary), and the caps had been sized against
the 768. A state at the caps where every payment carries an outcome (182 MB) ended
the process inside JSON.parse, and a payer could get there with far fewer payments
by padding its outcomes with 4,096-byte ERC-1271 signatures: about 24,000 of them
broke a restart. The checker imports outcomes, so both operators failed, an
out-of-memory abort is not an exception, so the .bak was never tried, and the
restart policy started each into the same failure.

- The file is read as bytes, outside V8's heap, and parsed a piece of at most
  1 MB at a time (parseJsonBuffer): a container larger than that is walked member
  by member, and runs of small members are parsed together. Same result as
  JSON.parse, refuses what it refuses, and the format is unchanged, so an older
  build reads what this one writes.
- Each agent's ledger is written in slices of 256, the same bytes as before. It
  was one string: 11 MB at the caps, 151 MB with padded outcomes.
- A payer outcome's signature is held to 1,024 bytes. The longest held on the
  primary on 15 Nov 2026 was 640, a smart account's. Both operators change this
  together, or one refuses imports the other accepted.

Measured under a 432 MB heap, synthetic state at the caps: today's mix (157 MB)
loads with 207 MB of heap instead of 346 MB, in 2.0 s instead of 1.2 s; every
payment with an outcome (182 MB) loads with 220 MB and 458 MB RSS, where it ran
out of heap before.

Not changed: the caps still count payments, not bytes, so a ledger filled to
150,000 payments that all carry 1,024-byte signatures is still too large for the
heap. That takes about 150,000 payments from an ERC-1271 payer and needs caps that
weigh bytes, which changes what both operators drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d count 1514 oracle tests

- scoreStandingsPending is declared and exported, as every metric the source
  increments must be.
- The answer-size tests' fake endpoint answers a JSON-RPC batch call by call. The
  payment-chain scan now asks for its blocks' times and code together, and ethers
  sends those as one batch; the real endpoint answers batches (measured).
- README and whitepaper: 1514 oracle tests, from 1500.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RunTimeAdmin
RunTimeAdmin merged commit c1af956 into main Nov 15, 2026
7 checks passed
@RunTimeAdmin
RunTimeAdmin deleted the fix/perf-review-critical branch November 15, 2026 17:54
RunTimeAdmin added a commit that referenced this pull request Nov 15, 2026
…fig [skip ci] (#313)

The limit live on archipelago-standby since 15 Nov 2026, with #311.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant