diff --git a/README.md b/README.md index c96341e..35100a6 100644 --- a/README.md +++ b/README.md @@ -601,10 +601,10 @@ Everything below runs offline, with no deployed contracts, except three fork sui | Suite | Count | Command | |---|---|---| | Contracts: unit, fuzz, E2E, upgrade, deploy-script, handover and 22 invariants | 799 | `forge test` | -| Oracle | 1514 | `cd oracle && node --test` | +| Oracle | 1532 | `cd oracle && node --test` | | SDK | 385 | `cd packages/sdk && npx vitest run` | -Counts are individual tests, so each of the 22 invariants counts once. The contracts were counted on 15 November 2026, the SDK on 15 November and the oracle on 14 November. Without a fork URL, 787 of the contract tests run and 12 skip. Run after `npm run build`, 376 of the SDK's 385 pass and 9 skip: the 8 integration tests, which need a live deployment, and one check that runs only in CI. Without a build, the 9 built-package tests skip too. Vitest strips types without checking them, so `npm run typecheck` in `packages/sdk` checks the tests against the SDK's types as well as the sources. [`scripts/mutation`](scripts/mutation/README.md) holds the harnesses that measure whether these tests would catch a deliberate break in the contracts or deploy scripts. +Counts are individual tests, so each of the 22 invariants counts once. The contracts were counted on 15 November 2026, the SDK on 15 November and the oracle on 15 November. Without a fork URL, 787 of the contract tests run and 12 skip. Run after `npm run build`, 376 of the SDK's 385 pass and 9 skip: the 8 integration tests, which need a live deployment, and one check that runs only in CI. Without a build, the 9 built-package tests skip too. Vitest strips types without checking them, so `npm run typecheck` in `packages/sdk` checks the tests against the SDK's types as well as the sources. [`scripts/mutation`](scripts/mutation/README.md) holds the harnesses that measure whether these tests would catch a deliberate break in the contracts or deploy scripts. Merges to `main` carry `[skip ci]`, to save CI minutes, so the push to `main` does not run the suites again. A change to code runs them on its pull request and merges only once every job there has passed; a docs-only change is checked locally with `node scripts/check-docs.mjs`. Each is checked against the `main` its branch started from, so two changes merged close together can still break `main`. That happened once, on 11 November 2026, and the next merge fixed it. The rule that every job must pass dates from 14 November 2026: before it, #281, #282 and #288 merged with their oracle job failing, and #283 and #297 fixed the tests that failed. On 13 November the oracle and SDK suites were run on `main` at `eaccb91` and passed. diff --git a/docs/whitepaper.md b/docs/whitepaper.md index 7dbc77d..7a97815 100644 --- a/docs/whitepaper.md +++ b/docs/whitepaper.md @@ -635,7 +635,7 @@ apply to a fresh deployment, not to the testnet. **Test coverage**, as of 15 November 2026: 799 contract tests across 36 Foundry test files including fuzz and invariant tests (the two slash drill suites fork Arc testnet themselves, and -the twelve in `SplitAuthorityFork` skip without `--fork-url`), 1514 oracle tests and 385 in the SDK (9 skip +the twelve in `SplitAuthorityFork` skip without `--fork-url`), 1532 oracle tests and 385 in the SDK (9 skip outside CI and without a live deployment; every test that runs passes). CI runs Foundry, the oracle and SDK suites, and Slither static analysis on every push, plus a documentation check that fails when the site states something the contracts do not. diff --git a/oracle/README.md b/oracle/README.md index 3e94f95..77b1ebf 100644 --- a/oracle/README.md +++ b/oracle/README.md @@ -131,6 +131,10 @@ answers 202 with `"durable": false` instead of 200 while a save is failing, and those as `accepted_not_durable`. Pulled payments are found again by the next scan; attested outcomes are on chain nowhere, so those do not come back. +A save writes the whole state and fsyncs it, about 2.8 s at the ledger's caps, so `/attest` writes that arrive +within 100 ms of each other share one, and each is answered after a save that holds it. +`sigvara_oracle_state_saves_total` counts saves. + `/flag`, `/flag/resolve` and `/link` say the same thing for the same reason: 202 with `"durable": false` while a save is failing, counted in `sigvara_oracle_writes_not_durable_total`. The change is held in memory and written by the @@ -472,6 +476,12 @@ evidence nobody can fetch is evidence nobody can audit. The response names `committedFields` — the exact fields the root commits to, in leaf order — so a verifier need not read this source to know what is covered. +The whole list comes in one body: about 32 MB for an agent at the cap of 25,000 payments, 41 MB with +payer outcomes published. `?offset=N&limit=M`, with M at most 1,000, serves part of it: the same fields, +then `offset`, `next` (where the next part starts, or null at the end) and those payments with their +proofs. A body is built a slice at a time, so other requests are answered while it is, and kept until the +agent's payments change. An agent with no payments is not kept. + Payments are served in the order the root is built over: by settlement time, then transaction and payer, with a payment chain's after the scored chain's. Two operators credit the same payments in different orders, and the order of the leaves can change the root, so the order they were credited in diff --git a/oracle/attest.test.js b/oracle/attest.test.js index ede8ea0..367a6da 100644 --- a/oracle/attest.test.js +++ b/oracle/attest.test.js @@ -228,6 +228,42 @@ test('a refused replay does not lock the payer out of the agent', async () => { assert.equal(next.status, 200, next.text); }); +// A save writes and fsyncs the whole state, about 2.8 s at the ledger's caps, and every accepted /attest had its own. +test('attestations that arrive together share a save, and each is answered after it', async () => { + const saves = async () => Number((await oracle.call('GET', '/metrics')).text.match(/^sigvara_oracle_state_saves_total (\d+)$/m)[1]); + const posts = Array.from({ length: 5 }, () => { + const agent = world.addAgent(); + return [agent.didHash, world.addTx([{ from: wallet(), to: agent.agentAddress, value: 5n * USDC }])]; + }); + const before = await saves(); + + const answers = await Promise.all(posts.map(([didHash, tx]) => attest(didHash, tx, { success: true, token: AUDITOR }))); + + for (const r of answers) assert.equal(r.status, 200, r.text); + const used = (await saves()) - before; + assert.ok(used >= 1 && used <= 2, `${used} saves for 5 attestations`); +}); + +// Anyone can copy a credited settlement from /evidence, and each replay used to cost six chain reads before its 409. +test('a replayed settlement is refused before the chain is read, and an auditor can still give it an outcome', async () => { + const agent = world.addAgent(); + const tx = world.addTx([{ from: wallet(), to: agent.agentAddress, value: 5n * USDC }]); + assert.equal((await attest(agent.didHash, tx, { success: true })).status, 200); + const receiptReads = () => oracle.calls().filter((c) => c.name === 'receipt' && c.args[0] === tx.toLowerCase()).length; + const before = receiptReads(); + + const replay = await attest(agent.didHash, tx.toUpperCase().replace('0X', '0x'), { success: true }); + assert.equal(replay.status, 409, replay.text); + assert.equal(replay.body.code, 'replayed'); + assert.equal(receiptReads(), before, 'refused without reading the payment'); + + // An outcome is something to add, so the auditor's report goes through the full check and is recorded. + const audited = await attest(agent.didHash, tx, { success: false, token: AUDITOR }); + assert.equal(audited.status, 200, audited.text); + assert.equal(audited.body.outcomeRecorded, true); + assert.equal(receiptReads(), before + 1); +}); + test('a stranger is held to the cooldown, and the refusal does not burn the receipt', async () => { const agent = world.addAgent(); const payer = wallet(); diff --git a/oracle/bond-watch.js b/oracle/bond-watch.js index b7ccabb..42ab277 100644 --- a/oracle/bond-watch.js +++ b/oracle/bond-watch.js @@ -7,8 +7,13 @@ // One is let back, and read at the next epoch like any other agent, when a log says its bond may have changed: a stake // deposited, a withdrawal, a slash or a status change naming it, and every one when the minimum stake changes. The logs // are read from where the last epoch's stopped, a little before it in case the node that answered the reads was behind -// the one that gave the head. Every agent is read on the first epoch, once a day after that, and whenever the logs or -// the head cannot be read, so a change no log showed waits a day at most. In memory only: a restart reads every agent. +// the one that gave the head. Every agent is read on the first epoch and whenever the logs or the head cannot be read. +// After that each one is read again within a day of the epoch that found it unbonded, so a change no log showed waits a +// day at most. In memory only: a restart reads every agent. +// +// Those reads are spread over the day's epochs, a share each, the agents found longest ago first. They used to come all +// at once: every agent read in one epoch, about 0.47 s each, which at 10,000 registrations (about 50 USDC of gas on Arc +// mainnet) is 78 minutes (performance review, 15 November). // // No dependencies, so it can be tested on its own. @@ -16,17 +21,40 @@ const SWEEP_MS = 24 * 3_600_000; const OVERLAP_BLOCKS = 100; function createBondWatch({ sweepMs = SWEEP_MS, overlapBlocks = OVERLAP_BLOCKS } = {}) { - const unbonded = new Set(); - // The head the logs were last read up to, and when every agent was last read. + // Each agent left out, with the time of the epoch that found it unbonded, the earliest first. + const unbonded = new Map(); + // The head the logs were last read up to. let cursor = null; - let sweptAtMs = null; + // When the last epoch began, and how many agents are owed a read so far: the unbonded share of the time since, + // carried over in fractions, so a few agents are not read every epoch for being rounded up. + let refreshedAtMs = null; + let owed = 0; const key = (didHash) => String(didHash).toLowerCase(); + function letEveryoneBack() { + unbonded.clear(); + owed = 0; + } + + // The share of the agents left out that the time since the last epoch is owed, and any found a day ago or more. + function letDueBack(nowMs, elapsedMs) { + owed += (unbonded.size * Math.max(0, elapsedMs)) / sweepMs; + for (const [didHash, foundAtMs] of unbonded) { + if (owed >= 1) owed -= 1; + else if (nowMs - foundAtMs < sweepMs) break; + unbonded.delete(didHash); + } + if (unbonded.size === 0) owed = 0; + } + return { /** Whether the epoch leaves this agent out of its reads. */ excluded: (didHash) => unbonded.has(key(didHash)), - /** An agent an epoch's read found below the minimum stake. */ - add(didHash) { unbonded.add(key(didHash)); }, + /** An agent an epoch's read found below the minimum stake, dated by that epoch. */ + add(didHash) { + unbonded.delete(key(didHash)); + unbonded.set(key(didHash), refreshedAtMs ?? 0); + }, size: () => unbonded.size, /** @@ -35,33 +63,35 @@ function createBondWatch({ sweepMs = SWEEP_MS, overlapBlocks = OVERLAP_BLOCKS } * epoch, with `error` when that is because a read failed. */ async refresh({ readHead, readChanges, nowMs }) { + const elapsedMs = refreshedAtMs === null ? 0 : nowMs - refreshedAtMs; + refreshedAtMs = nowMs; let head; try { head = await readHead(); } catch (error) { - unbonded.clear(); + letEveryoneBack(); cursor = null; return { full: true, error }; } - if (cursor === null || nowMs - sweptAtMs >= sweepMs) { - unbonded.clear(); + if (cursor === null) { + letEveryoneBack(); cursor = head; - sweptAtMs = nowMs; return { full: true }; } + letDueBack(nowMs, elapsedMs); const from = Math.max(0, cursor - overlapBlocks + 1); cursor = Math.max(cursor, head); if (head < from) return { full: false }; try { const changes = await readChanges(from, head); if (changes.minimumChanged) { - unbonded.clear(); + letEveryoneBack(); return { full: true }; } for (const didHash of changes.agents) unbonded.delete(key(didHash)); return { full: false }; } catch (error) { - unbonded.clear(); + letEveryoneBack(); return { full: true, error }; } }, diff --git a/oracle/chain.js b/oracle/chain.js index 2d6faa3..6b58d37 100644 --- a/oracle/chain.js +++ b/oracle/chain.js @@ -14,7 +14,8 @@ const STAKE_VIEW_ABI = ['function hasMinimumStake(bytes32 didHash) view returns const { SCORE_FACTORS } = require('./epoch-policy'); const { readSplitting, spanRefused } = require('./log-spans'); -const { isTransientReadError, readWithBackoff, MAX_BATCH_CALLS } = require('./read-retry'); +const { isTransientReadError, readWithBackoff } = require('./read-retry'); +const { rpcProvider } = require('./rpc-provider'); const reviewers = require('./reviewers'); const { createFunderReader } = require('./funder-scan'); @@ -182,7 +183,7 @@ function init(cfg, deps = {}) { cfg_ = cfg; txWaitMs = cfg.txWaitMs ?? DEFAULT_TX_WAIT_MS; txPollMs = cfg.txPollMs ?? DEFAULT_TX_POLL_MS; - provider = deps.provider ?? new ethers.JsonRpcProvider(cfg.rpcUrl, undefined, { batchMaxCount: MAX_BATCH_CALLS }); + provider = deps.provider ?? rpcProvider(cfg.rpcUrl, cfg.expectedChainId); wallet = deps.wallet ?? new CappedWallet(cfg.privateKey, provider, capsFromConfig(cfg)); identityContract = deps.identityContract ?? new ethers.Contract(cfg.identityAddress, IDENTITY_ABI, provider); reputationContract = deps.reputationContract ?? new ethers.Contract(cfg.reputationAddress, REPUTATION_ABI, wallet); diff --git a/oracle/credit-reread.test.js b/oracle/credit-reread.test.js index 595ee51..605e979 100644 --- a/oracle/credit-reread.test.js +++ b/oracle/credit-reread.test.js @@ -121,7 +121,7 @@ const ev = (i, ts, over = {}) => Object.freeze({ const endsAt = (e) => (e.removedAt != null ? Number(e.removedAt) : Infinity); const NO_CHAIN = paymentChains.readConfig({}, { scoredChainId: 5042002 }); -test('a root made before a removal still reproduces from the state that holds it, and the next one leaves it out', () => { +test('a root made before a removal still reproduces from the state that holds it, and the next one leaves it out', async () => { const a = ev(1, NOW - 60 * MIN); const b = ev(2, NOW - 50 * MIN); const before = rootFor([a, b]); // proposed at NOW - 30 min @@ -133,16 +133,16 @@ test('a root made before a removal still reproduces from the state that holds it assert.equal(rootFor(paymentChains.eventsAt(held, NOW, NO_CHAIN)), rootFor([a])); // The checker auditing each proposal from the same state reaches the same answer. - const audit = (atMs, root, cuts) => evidenceAsOf( + const audit = async (atMs, root, cuts) => evidenceAsOf( paymentChains.eventsAt(held, atMs, NO_CHAIN, undefined, { keepRemoved: true }), atMs, root, { endsAt, ...(cuts ? { cuts } : {}) }, ); - assert.equal(audit(NOW - 30 * MIN, before).matchedRoot, true, 'the earlier proposal'); - assert.equal(audit(NOW, rootFor([a])).matchedRoot, true, 'the later one'); - assert.equal(audit(NOW, before).matchedRoot, false, 'a proposal after the removal that still counts it is a disagreement'); + assert.equal((await audit(NOW - 30 * MIN, before)).matchedRoot, true, 'the earlier proposal'); + assert.equal((await audit(NOW, rootFor([a]))).matchedRoot, true, 'the later one'); + assert.equal((await audit(NOW, before)).matchedRoot, false, 'a proposal after the removal that still counts it is a disagreement'); // With payment chains on, a proposal dated after the removal whose evidence was cut before it still counted it. const cuts = [NOW - 19 * MIN, NOW - 20 * MIN, NOW - 21 * MIN]; - assert.equal(audit(NOW - 18 * MIN, before, cuts).matchedRoot, true); + assert.equal((await audit(NOW - 18 * MIN, before, cuts)).matchedRoot, true); }); test('the scan canary never points at a payment a reorganization removed', () => { diff --git a/oracle/evidence-as-of.js b/oracle/evidence-as-of.js index 70f5a7f..b050d62 100644 --- a/oracle/evidence-as-of.js +++ b/oracle/evidence-as-of.js @@ -41,22 +41,39 @@ const merkle = require('./merkle'); * * Only the fixed order is read (payment-chains.eventsAt): it is the order every proposal commits to. * + * Every tree tried shares one memo of pair hashes (merkle.pairOf). Neighbouring cut-offs hold nearly the same payments + * in the same order, so their trees share every pair left of where they differ, and each was built whole: a miss at + * the cap of 25,000 payments held the event loop for 22 s (performance review, 15 November). A cut-off that counts + * the same payments as the one before is skipped, and other work runs between trees. + * * @param {object[]} events one agent's payment events, in the fixed order * @param {number} atMs the proposal's time, in ms * @param {string|null} committedRoot the proposal's evidenceRoot, or null when it has none * @param {{ eligibleAt?: (e: object) => number, endsAt?: (e: object) => number, cuts?: number[] }} [options] - * @returns {{ events: object[], matchedRoot: boolean }} + * @returns {Promise<{ events: object[], matchedRoot: boolean }>} */ -function evidenceAsOf(events, atMs, committedRoot, { eligibleAt = (e) => Number(e.ts), endsAt = () => Infinity, cuts = [] } = {}) { +async function evidenceAsOf(events, atMs, committedRoot, { eligibleAt = (e) => Number(e.ts), endsAt = () => Infinity, cuts = [] } = {}) { if (!Number.isFinite(atMs)) throw new TypeError('evidenceAsOf: atMs (ms) is required'); const countedBy = (list, cut) => list.filter((e) => eligibleAt(e) <= cut && endsAt(e) > cut); const committed = typeof committedRoot === 'string' ? committedRoot.toLowerCase() : null; if (committed) { + const pairs = new Map(); + // One unset copy of each event, so its leaf is hashed once however many cut-offs hold it. + const unsetCopies = new Map(); + const unsetOf = (e) => { + if (!signedAfter(e, atMs)) return e; + if (!unsetCopies.has(e)) unsetCopies.set(e, withoutPayerOutcome(e)); + return unsetCopies.get(e); + }; + let last = null; for (const cut of [atMs, ...cuts.filter((c) => c < atMs)]) { const set = countedBy(events, cut); - if (merkle.rootFor(set).toLowerCase() === committed) return { events: set, matchedRoot: true }; - const unset = set.map((e) => (signedAfter(e, atMs) ? withoutPayerOutcome(e) : e)); - if (unset.some((e, i) => e !== set[i]) && merkle.rootFor(unset).toLowerCase() === committed) { + if (last && set.length === last.length && set.every((e, i) => e === last[i])) continue; + if (last) await new Promise((resolve) => setImmediate(resolve)); + last = set; + if (merkle.rootFor(set, pairs).toLowerCase() === committed) return { events: set, matchedRoot: true }; + const unset = set.map(unsetOf); + if (unset.some((e, i) => e !== set[i]) && merkle.rootFor(unset, pairs).toLowerCase() === committed) { return { events: unset, matchedRoot: true }; } } diff --git a/oracle/evidence-as-of.test.js b/oracle/evidence-as-of.test.js index 028f3e4..7c4cbbc 100644 --- a/oracle/evidence-as-of.test.js +++ b/oracle/evidence-as-of.test.js @@ -17,54 +17,54 @@ const ev = (i, tsMs, over = {}) => Object.freeze({ ...over, }); -test('keeps only what had settled by the proposal, and names the committed root when it is that', () => { +test('keeps only what had settled by the proposal, and names the committed root when it is that', async () => { const before = [ev(1, AT - 86_400_000), ev(2, AT)]; const after = [ev(3, AT + 1), ev(4, AT + 3_600_000)]; - const got = evidenceAsOf([...before, ...after], AT, rootFor(before)); + const got = await evidenceAsOf([...before, ...after], AT, rootFor(before)); assert.deepEqual(got.events, before); assert.equal(got.matchedRoot, true); }); -test('finds the proposal\'s evidence when a payer signed an outcome after it', () => { +test('finds the proposal\'s evidence when a payer signed an outcome after it', async () => { // Settled before the proposal with no outcome; the payer signed one an hour after it. const unjudged = ev(1, AT - 60_000, { success: null }); const signed = Object.freeze({ ...unjudged, success: true, payerOutcome: Object.freeze({ outcome: 1, acceptedAt: AT + 3_600_000 }) }); const old = ev(2, AT - 86_400_000); - const got = evidenceAsOf([old, signed], AT, rootFor([old, unjudged])); + const got = await evidenceAsOf([old, signed], AT, rootFor([old, unjudged])); assert.equal(got.matchedRoot, true); assert.equal(got.events[1].success, null, 'the outcome is read as unreported, as the proposer saw it'); assert.equal(got.events[1].payerOutcome, undefined); assert.equal(rootFor(got.events), rootFor([old, unjudged])); }); -test('keeps an outcome the payer signed before the proposal', () => { +test('keeps an outcome the payer signed before the proposal', async () => { const signed = ev(1, AT - 60_000, { payerOutcome: Object.freeze({ outcome: 1, acceptedAt: AT - 1 }) }); - const got = evidenceAsOf([signed], AT, rootFor([signed])); + const got = await evidenceAsOf([signed], AT, rootFor([signed])); assert.equal(got.matchedRoot, true); assert.equal(got.events[0], signed); }); -test('when no reading matches the committed root, uses what had settled, outcomes as held', () => { +test('when no reading matches the committed root, uses what had settled, outcomes as held', async () => { const held = [ev(1, AT - 60_000), ev(2, AT - 120_000, { payerOutcome: Object.freeze({ acceptedAt: AT + 5 }) })]; const elsewhere = rootFor([ev(9, AT - 60_000)]); - const got = evidenceAsOf([...held, ev(3, AT + 10)], AT, elsewhere); + const got = await evidenceAsOf([...held, ev(3, AT + 10)], AT, elsewhere); assert.equal(got.matchedRoot, false); assert.deepEqual(got.events, held); // No committed root at all reads the same way. - assert.deepEqual(evidenceAsOf([...held, ev(3, AT + 10)], AT, null), { events: held, matchedRoot: false }); + assert.deepEqual(await evidenceAsOf([...held, ev(3, AT + 10)], AT, null), { events: held, matchedRoot: false }); }); -test('matches a proposal with no evidence, and compares roots without case', () => { - assert.deepEqual(evidenceAsOf([ev(1, AT + 1)], AT, ethers.ZeroHash), { events: [], matchedRoot: true }); +test('matches a proposal with no evidence, and compares roots without case', async () => { + assert.deepEqual(await evidenceAsOf([ev(1, AT + 1)], AT, ethers.ZeroHash), { events: [], matchedRoot: true }); const one = [ev(1, AT - 1)]; - assert.equal(evidenceAsOf(one, AT, rootFor(one).toUpperCase().replace('0X', '0x')).matchedRoot, true); + assert.equal((await evidenceAsOf(one, AT, rootFor(one).toUpperCase().replace('0X', '0x'))).matchedRoot, true); }); -test('refuses a time that is not one', () => { - assert.throws(() => evidenceAsOf([], undefined, null), /atMs/); +test('refuses a time that is not one', async () => { + await assert.rejects(evidenceAsOf([], undefined, null), /atMs/); }); -test('only the fixed order is read: a root over the same payments in another order does not match', () => { +test('only the fixed order is read: a root over the same payments in another order does not match', async () => { const unjudged = ev(1, AT - 60_000, { success: null }); const signed = Object.freeze({ ...unjudged, success: true, payerOutcome: Object.freeze({ outcome: 1, acceptedAt: AT + 3_600_000 }) }); const a = ev(2, AT - 86_400_000); @@ -73,8 +73,55 @@ test('only the fixed order is read: a root over the same payments in another ord const root = rootFor([unjudged, a, b]); assert.notEqual(rootFor([a, b, unjudged]), root, 'the order changes this root'); - const got = evidenceAsOf(fixed, AT, root); + const got = await evidenceAsOf(fixed, AT, root); assert.equal(got.matchedRoot, false); assert.deepEqual(got.events, fixed, 'what had settled by then, in the fixed order'); assert.equal(got.events[2].success, true); }); + +// ---- cost ---------------------------------------------------------------------------------------------------------- + +const many = (n) => Array.from({ length: n }, (_, i) => Object.freeze({ + txHash: '0x' + (i + 1).toString(16).padStart(64, '0'), + payer: '0x' + '11'.repeat(20), + amount: '1000', + ts: AT - (n - i) * 1000, + success: true, +})); + +/** How many keccak256 hashes `fn` computes. merkle.js reads ethers.keccak256 at each call, so it can be counted. */ +async function hashesIn(fn) { + const real = ethers.keccak256; + let n = 0; + Object.defineProperty(ethers, 'keccak256', { value: (...a) => { n += 1; return real(...a); }, configurable: true, writable: true }); + try { + await fn(); + } finally { + Object.defineProperty(ethers, 'keccak256', { value: real, configurable: true, writable: true }); + } + return n; +} + +// Every cut-off's tree was built whole: an audit that matched none at the cap of 25,000 payments held the event loop for +// 22 s (performance review, 15 November). Neighbouring cut-offs share every pair left of where they differ. +test('the trees of neighbouring cut-offs share their pairs', async () => { + const N = 3000; + const events = many(N); + rootFor(events); // the leaves, hashed once and kept, so only the pairs are counted below + const cuts = Array.from({ length: 30 }, (_, k) => AT - (k + 1) * 60_000); + let got; + const hashes = await hashesIn(async () => { got = await evidenceAsOf(events, AT, ethers.ZeroHash.replace(/0$/, '1'), { cuts }); }); + assert.equal(got.matchedRoot, false); + // Built whole, the 31 trees take about 31 times N. Shared, about N and a few dozen for each further cut-off. + assert.ok(hashes < 2 * N, `${hashes} hashes for ${cuts.length + 1} cut-offs over ${N} payments`); +}); + +test('an audit lets other work run between the trees it tries', async () => { + const events = many(50); + const cuts = Array.from({ length: 5 }, (_, k) => AT - (k + 1) * 5_000); + let ran = false; + const audit = evidenceAsOf(events, AT, ethers.ZeroHash.replace(/0$/, '1'), { cuts }); + setImmediate(() => { ran = true; }); + await audit; + assert.equal(ran, true); +}); diff --git a/oracle/evidence-body.test.js b/oracle/evidence-body.test.js new file mode 100644 index 0000000..b50ef2c --- /dev/null +++ b/oracle/evidence-body.test.js @@ -0,0 +1,95 @@ +'use strict'; + +// GET /evidence/:didHash end to end: the whole body, its pages, and what the cache keeps +// (performance review, 15 November). + +const { test, before, after } = require('node:test'); +const assert = require('node:assert/strict'); +const merkle = require('./merkle'); +const { World, startOracle } = require('./test-support/oracle-harness'); + +const N = 600; +let world; +let oracle; +let agent; + +before(async () => { + world = new World(); + agent = world.addRegisteredAgent({ registeredDaysAgo: 30 }); + const nowMs = world.data.nowSec * 1000; + const events = Array.from({ length: N }, (_, i) => ({ + txHash: `0x${(i + 1).toString(16).padStart(64, '0')}`, + ts: nowMs - (N - i) * 60_000, + amount: '5000000', + payer: `0x${String((i % 7) + 1).repeat(40)}`, + success: i % 3 === 0 ? null : true, + })); + // No payment verification, so the epoch reads none of these payments again: they are on no chain here. + oracle = await startOracle({ world, files: { 'state.json': JSON.stringify({ schemaVersion: 1, paymentEvents: { [agent.didHash]: events } }) } }); + // Nothing below may meet the startup epoch changing the list. + await oracle.waitForLog(/epoch done/); +}); + +after(async () => { + await oracle.stop(); +}); + +const get = (query = '') => oracle.call('GET', `/evidence/${agent.didHash}${query}`); + +test('the whole body is built a slice at a time, and is the bytes JSON.stringify gives', async () => { + const r = await get(); + assert.equal(r.status, 200, r.text); + assert.equal(JSON.stringify(JSON.parse(r.text)), r.text); + assert.equal(r.body.count, N); + assert.equal(r.body.evidence.length, N); + for (const i of [0, 255, 256, 599]) { + assert.ok(merkle.verifyProof(r.body.evidence[i].leaf, r.body.evidence[i].proof, r.body.evidenceRoot), `proof ${i}`); + } +}); + +// The body is every payment with its proof: 31.6 MB for an agent at the cap of 25,000, 41.2 MB with payer outcomes. +test('pages of the body, asked for by offset and limit, put together are the whole list', async () => { + const whole = (await get()).body; + const pages = []; + for (let offset = 0; offset !== null;) { + const r = await get(`?offset=${offset}&limit=250`); + assert.equal(r.status, 200, r.text); + assert.equal(r.body.offset, offset); + assert.equal(r.body.evidenceRoot, whole.evidenceRoot); + assert.equal(r.body.count, N); + pages.push(r.body); + offset = r.body.next; + } + assert.deepEqual(pages.map((p) => p.next), [250, 500, null]); + assert.deepEqual(pages.flatMap((p) => p.evidence), whole.evidence); + for (const bad of ['?limit=0', '?limit=1001', '?offset=-1', '?offset=1.5', '?offset=abc']) { + assert.equal((await get(bad)).status, 400, bad); + } +}); + +// One agent's hash has many spellings, and each used to be its own entry and its own Merkle rebuild. +test('the cache is keyed on the stored spelling, not the caller\'s', async () => { + const metric = async (re) => Number((await oracle.call('GET', '/metrics')).text.match(re)[1]); + const ENTRIES = /^sigvara_oracle_evidence_cache_entries (\d+)$/m; + const HITS = /^sigvara_oracle_evidence_cache_total\{result="hit"\} (\d+)$/m; + await get(); + const [entries, hits] = [await metric(ENTRIES), await metric(HITS)]; + const upper = `0x${agent.didHash.slice(2).toUpperCase()}`; + const mixed = `0x${agent.didHash.slice(2, 10).toUpperCase()}${agent.didHash.slice(10)}`; + for (const spelling of [upper, mixed]) assert.equal((await oracle.call('GET', `/evidence/${spelling}`)).status, 200); + assert.equal(await metric(ENTRIES), entries, 'no entry of their own'); + assert.equal(await metric(HITS), hits + 2, 'both served from the one there is'); +}); + +// One body stored for every didHash anyone asked about pushed out the bodies worth keeping. +test('an agent with nothing to list is not cached', async () => { + const entries = async () => Number((await oracle.call('GET', '/metrics')).text.match(/^sigvara_oracle_evidence_cache_entries (\d+)$/m)[1]); + await get(); + const held = await entries(); + for (let i = 0; i < 3; i += 1) { + const r = await oracle.call('GET', `/evidence/0x${String(i + 1).repeat(64)}`); + assert.equal(r.status, 200, r.text); + assert.equal(r.body.count, 0); + } + assert.equal(await entries(), held); +}); diff --git a/oracle/explorer-merkle.test.js b/oracle/explorer-merkle.test.js index 71eafdb..8d3a724 100644 --- a/oracle/explorer-merkle.test.js +++ b/oracle/explorer-merkle.test.js @@ -68,11 +68,11 @@ test('a payment the oracle could not commit to is refused, not hashed', () => { assert.throws(() => explorer.leafOf({ ...e, amount: '-1' })); }); -test('matchRoot finds the set a proposal committed to, as the checker does', () => { +test('matchRoot finds the set a proposal committed to, as the checker does', async () => { const events = Object.freeze(Array.from({ length: 7 }, (_, i) => stored(i))); // A proposal made after the fourth payment settled. const atMs = events[3].ts + 30_000; - const asOf = evidenceAsOf(events, atMs, null); + const asOf = await evidenceAsOf(events, atMs, null); const committed = merkle.rootFor(asOf.events); const list = events.map(served); @@ -89,7 +89,7 @@ test('matchRoot finds the set a proposal committed to, as the checker does', () assert.equal(explorer.matchRoot(list, merkle.rootFor(events)).events.length, 7); }); -test('matchRoot reads a payer-signed outcome taken after the proposal as unreported', () => { +test('matchRoot reads a payer-signed outcome taken after the proposal as unreported', async () => { const at = (T0 + 10 * 60) * 1000; const late = { acceptedAt: at + 3_600_000 }; const events = Object.freeze([ @@ -98,7 +98,7 @@ test('matchRoot reads a payer-signed outcome taken after the proposal as unrepor stored(2), ]); // What the proposer had: the signed outcome did not exist yet. - const asOf = evidenceAsOf(events, at, merkle.rootFor(events.map((e) => (e.payerOutcome ? Object.freeze({ ...e, success: null, payerOutcome: undefined }) : e)))); + const asOf = await evidenceAsOf(events, at, merkle.rootFor(events.map((e) => (e.payerOutcome ? Object.freeze({ ...e, success: null, payerOutcome: undefined }) : e)))); assert.equal(asOf.matchedRoot, true); const committed = merkle.rootFor(asOf.events); diff --git a/oracle/external.js b/oracle/external.js index e2344d6..6ed493d 100644 --- a/oracle/external.js +++ b/oracle/external.js @@ -25,7 +25,8 @@ const { ethers } = require('ethers'); const { CAPS } = require('./score-caps'); -const { readWithBackoff, MAX_BATCH_CALLS } = require('./read-retry'); +const { readWithBackoff } = require('./read-retry'); +const { rpcProvider } = require('./rpc-provider'); const { accountRefused: registryRefused } = require('./payer-outcome'); const ID_ABI = [ @@ -122,7 +123,7 @@ let checkedAt = 0; function init(cfg, deps = {}) { enabled = !!(cfg.externalRpc && cfg.externalIdentity && cfg.externalReputation); if (!enabled) return; - provider = deps.provider ?? new ethers.JsonRpcProvider(cfg.externalRpc, undefined, { batchMaxCount: MAX_BATCH_CALLS }); + provider = deps.provider ?? rpcProvider(cfg.externalRpc); idContract = deps.idContract ?? new ethers.Contract(cfg.externalIdentity, ID_ABI, provider); repContract = deps.repContract ?? new ethers.Contract(cfg.externalReputation, REP_ABI, provider); // A re-init points at a different chain, so anything cached about the old one is a @@ -152,8 +153,8 @@ async function refreshChainId(maxAgeMs = 0, nowMs = Date.now()) { async function checkChainId() { try { - const net = await withTimeout(provider.getNetwork()); - chainId = Number(net.chainId); + // Asked of the node each time: the provider's own network is fixed at its first answer (rpc-provider.js). + chainId = Number(BigInt(await withTimeout(provider.send('eth_chainId', [])))); chainIdError = null; } catch (err) { chainId = null; diff --git a/oracle/external.test.js b/oracle/external.test.js index bb43945..a4cdd69 100644 --- a/oracle/external.test.js +++ b/oracle/external.test.js @@ -115,7 +115,7 @@ test('chainState: disabled when the feed is not configured', () => { test('chainState: configured but unreachable is NOT reported as configured', async () => { // The whole point. Three non-empty strings used to read as success. external.init(CFG, { - provider: { getNetwork: async () => { throw new Error('ECONNREFUSED'); } }, + provider: { send: async () => { throw new Error('ECONNREFUSED'); } }, idContract: {}, repContract: {}, }); await external.refreshChainId(); @@ -128,7 +128,7 @@ test('chainState: configured but unreachable is NOT reported as configured', asy test('chainState: reports the chain it actually reached', async () => { external.init(CFG, { - provider: { getNetwork: async () => ({ chainId: 5042002n }) }, + provider: { send: async () => '0x4cef52' }, idContract: {}, repContract: {}, }); await external.refreshChainId(); @@ -140,14 +140,14 @@ test('chainState: a cached id from the old chain does not survive a re-init', as // Re-init means the feed was repointed. Reporting the previous chain afterwards is // precisely the failure this was written to make visible. external.init(CFG, { - provider: { getNetwork: async () => ({ chainId: 84532n }) }, + provider: { send: async () => '0x14a34' }, idContract: {}, repContract: {}, }); await external.refreshChainId(); assert.strictEqual(external.chainState().chainId, 84532); external.init(CFG, { - provider: { getNetwork: async () => ({ chainId: 5042002n }) }, + provider: { send: async () => '0x4cef52' }, idContract: {}, repContract: {}, }); assert.strictEqual(external.chainState().status, 'unreachable', 'stale id must be dropped immediately'); @@ -158,7 +158,7 @@ test('chainState: a cached id from the old chain does not survive a re-init', as test('refreshChainId: never throws, whatever the provider does', async () => { external.init(CFG, { - provider: { getNetwork: async () => { throw new Error('boom'); } }, + provider: { send: async () => { throw new Error('boom'); } }, idContract: {}, repContract: {}, }); assert.strictEqual(await external.refreshChainId(), null); @@ -169,7 +169,7 @@ test('refreshChainId: never throws, whatever the provider does', async () => { test('refreshChainId: a hung provider does not hang the caller forever', async () => { external.init(CFG, { - provider: { getNetwork: () => new Promise(() => {}) }, // never settles + provider: { send: () => new Promise(() => {}) }, // never settles idContract: {}, repContract: {}, }); const started = Date.now(); @@ -185,7 +185,7 @@ test('refreshChainId: a second caller waits for the check in flight', async () = let release; let calls = 0; external.init(CFG, { - provider: { getNetwork: () => { calls += 1; return new Promise((r) => { release = () => r({ chainId: 5042002n }); }); } }, + provider: { send: () => { calls += 1; return new Promise((r) => { release = () => r('0x4cef52'); }); } }, idContract: {}, repContract: {}, }); const first = external.refreshChainId(); @@ -200,7 +200,7 @@ test('refreshChainId: a cached id is kept for maxAgeMs, then checked again, and let up = true; let calls = 0; external.init(CFG, { - provider: { getNetwork: async () => { calls += 1; if (!up) throw new Error('ECONNREFUSED'); return { chainId: 5042002n }; } }, + provider: { send: async () => { calls += 1; if (!up) throw new Error('ECONNREFUSED'); return '0x4cef52'; } }, idContract: {}, repContract: {}, }); await external.refreshChainId(); diff --git a/oracle/http.test.js b/oracle/http.test.js index 3848216..70f2ff8 100644 --- a/oracle/http.test.js +++ b/oracle/http.test.js @@ -462,20 +462,24 @@ test('a body far past the limit is cut off, and the oracle carries on', async () // ---- the evidence cache ---------------------------------------------------------------- -test('the evidence cache is keyed on the stored spelling, not the caller\'s', async () => { +test('an agent with no payments is served the same under every spelling, and none is cached', async () => { const lower = `0x${'ef'.repeat(32)}`; const upper = `0x${'EF'.repeat(32)}`; const mixed = `0x${'Ef'.repeat(32)}`; const entriesBefore = counter(await metricsText(), 'sigvara_oracle_evidence_cache_entries'); + const bodies = []; for (const spelling of [lower, upper, mixed]) { - assert.equal((await oracle.call('GET', `/evidence/${spelling}`)).status, 200); + const r = await oracle.call('GET', `/evidence/${spelling}`); + assert.equal(r.status, 200); + bodies.push(r.text); } - // Entries, not hits: an agent with no payments gets a fresh empty array on every read, and - // the cache treats the array's identity as the revision, so it never hits for such an agent. - assert.equal(counter(await metricsText(), 'sigvara_oracle_evidence_cache_entries'), entriesBefore + 1, - 'three spellings of one agent are one cache entry'); + assert.equal(new Set(bodies).size, 1, 'one body, under the stored spelling'); + assert.equal(JSON.parse(bodies[0]).didHash, lower); + // One stored for every didHash anyone asked about pushed out the bodies worth keeping. The keying is checked on + // an agent with payments in evidence-body.test.js. + assert.equal(counter(await metricsText(), 'sigvara_oracle_evidence_cache_entries'), entriesBefore); }); // ---- credential scopes ----------------------------------------------------------------- diff --git a/oracle/index.js b/oracle/index.js index 399d58e..305b955 100644 --- a/oracle/index.js +++ b/oracle/index.js @@ -339,7 +339,7 @@ const { resolveFlags, links, load: loadState, - persist: persistState, + persist: saveState, checkAttestCooldown, recordAttestation, creditPayment, @@ -388,6 +388,12 @@ const { ATTEST_COOLDOWN_MS, } = require('./store'); +/** Writes the whole state (store.persist), counted for /metrics. Returns whether it landed. */ +function persistState() { + metrics.inc('stateSaves'); + return saveState(); +} + // No settlement older than pruning keeps is credited, by any path: the same half-life the epoch prunes by. setCreditAgeLimit(paymentCfg.halfLifeMs); // A full ledger drops its smallest payments first, measured in fee units as the score measures them. @@ -426,6 +432,9 @@ function operatorNamedAt(didHash) { // Nobody named: every payment that counts by its time, whoever paid it. const NOBODY_NAMED = () => undefined; +// A transaction hash as a caller may send it, in either case. +const TX_HASH = /^0x[0-9a-fA-F]{64}$/; + /** * How the checker reads a proposal's evidence (evidence-as-of.js). With a payment chain on, at each * whole-minute cut-off in the half hour before it (payment-chains.auditCuts), since the proposer chose its @@ -2069,7 +2078,7 @@ async function runEpochInner() { // Whoever paid, and removed or not: a payer named the operator, or a payment removed, after the proposer's // cut-off still counted for it, so each cut-off is judged on its own (auditOptions). asOf = payments.required(paymentCfg) - ? evidenceAsOf(evidenceAt(didHash, atProposalMs, NOBODY_NAMED, { keepRemoved: true }), atProposalMs, freshPending.evidenceRoot ?? null, + ? await evidenceAsOf(evidenceAt(didHash, atProposalMs, NOBODY_NAMED, { keepRemoved: true }), atProposalMs, freshPending.evidenceRoot ?? null, auditOptions(atProposalMs, operatorNamedAt(didHash))) : null; // The external factor too, as the primary would have read it at its proposal: same @@ -2485,6 +2494,27 @@ async function runEpochInner() { pruneExpiredCooldowns(); } +/** + * A save shared by the writes that arrive together, resolving to whether it landed. + * + * A save writes the whole state and fsyncs it: about 2.8 s at the ledger's caps (performance review, 15 November). + * With one for every accepted /attest, a steady stream of them kept this oracle saving and fell behind. Each write + * still answers only after a save that includes it, so a 200 is as durable as it was. The token-only routes (/flag, + * /link) keep their own save: their replies are remembered for a retry, and a retry must not land between a change + * and its save. + */ +const SAVE_WINDOW_MS = 100; +let nextSave = null; +function saveSoon() { + nextSave ??= new Promise((resolve) => { + setTimeout(() => { + nextSave = null; + resolve(persistState()); + }, SAVE_WINDOW_MS); + }); + return nextSave; +} + // ---- HTTP API -------------------------------------------------------------- // 32MB of serialized evidence responses. The revision token is the payment-event array @@ -2787,14 +2817,14 @@ async function admitPayerOutcome({ didHash, credited, info, signed, packetId = n trimLedger(didHash); if (filed === 'already_set' || filed === 'outside_window') { // Payments credited above stay credited; keep them through a restart. - if (credits > 0 && !fromPeer) persistState(); + if (credits > 0 && !fromPeer) await saveSoon(); const r = refuse(409, 'outcome_recorded', filed === 'already_set' ? 'the payer has already signed an outcome for this payment' : 'an outcome recorded for this payment is final', 'attestRejectedPayment'); return { ...r, body: { ...r.body, ...heldOutcomeReply(didHash, credited.txHash, signed, chainId) } }; } if (filed === 'not_found') { - if (credits > 0 && !fromPeer) persistState(); + if (credits > 0 && !fromPeer) await saveSoon(); const r = isDropped(credited.txHash, didHash, leg.payer, chainId) ? refuse(409, 'payment_dropped', PAYMENT_DROPPED, 'attestRejectedPayment') : refuse(409, 'replayed', 'this settlement has already been credited', 'attestRejectedPayment'); @@ -2811,14 +2841,16 @@ async function admitPayerOutcome({ didHash, credited, info, signed, packetId = n if (delivered) att.successful++; } attestations.set(didHash, att); - const durable = fromPeer ? true : persistState(); + // The tally as this outcome left it, since others can move it while the save is waited for. + const tally = { ...att }; + const durable = fromPeer ? true : await saveSoon(); return { status: durable ? 200 : 202, metric: durable ? 'attestAccepted' : 'attestAcceptedNotDurable', body: { didHash, attester: leg.payer, - ...att, + ...tally, ...(durable ? {} : { durable: false }), payment: { txHash: credited.txHash, amount: leg.amount.toString(), ...(pc ? { chainId } : {}) }, outcomeRecorded: true, @@ -3105,18 +3137,11 @@ function outcomeSource(event) { return typeof event.success === 'boolean' ? 'credentialed' : null; } -/** - * The /evidence body for an agent, as a JSON string. - * - * Built once per (agent, payment set). Everything here is a pure function of `events`, - * which is why it caches cleanly: the tree, every proof and the serialization are - * O(E log E) work that does not change until a payment is credited or pruned. - */ -function buildEvidenceBody(didHash, events) { - const tree = merkle.buildTree(events); - return JSON.stringify({ +/** What the /evidence body says before its list of payments: the same on every page of an agent's evidence. */ +function evidenceHead(didHash, events, root) { + return { didHash, - evidenceRoot: tree.root, + evidenceRoot: root, count: events.length, // Exactly what the root commits to, in leaf order, so nobody has to read this // service's source to know which fields are covered. counterauditPacketId is @@ -3151,40 +3176,87 @@ function buildEvidenceBody(didHash, events) { // proof, and says so in the response. For a payer-signed outcome it can be checked: // CounterAudit's public proof names the EIP-712 hash of the outcome it sealed. unverifiedFields: ['counterauditPacketId'], - // The leaf is derivable from the payment, so a verifier rebuilds it rather than - // trusting the one served here; it is included to make that comparison easy. - evidence: events.map((e, i) => ({ - ...(e.chainId != null ? { chainId: e.chainId } : {}), - txHash: e.txHash ?? null, - payer: e.payer, - amount: e.amount, - settledAt: merkle.settledSeconds(e), - success: e.success, - // Who the outcome is the word of (docs/adr/0006-payer-signed-outcomes.md): the payer, by a - // signature anyone can check, or a credential holder. null when there is none. - outcomeSource: outcomeSource(e), - // A payer's "not delivered" (ADR 0011): when it counts from, and the refund that cured it if one did. - // `success` above is how it reads in this evidence, null until it counts. Not part of the leaf. - ...(e.payerOutcome?.outcome === negativeHold.NOT_DELIVERED ? { - payerNegative: { - countsFrom: Number.isFinite(negativeHold.countsFromMs(e, negativeHoldCfg)) - ? new Date(negativeHold.countsFromMs(e, negativeHoldCfg)).toISOString() : null, - settled: e.refundCheck?.final === true, - refundTx: e.refundCheck?.txHash ?? null, - }, - } : {}), - leaf: tree.leaves[i], - proof: merkle.proofFor(tree, i), - // Not part of the leaf and not verified here. See committedFields and unverifiedFields above. - ...(e.packetId ? { counterauditPacketId: e.packetId } : {}), - // The payer's signed payload, so a verifier can check that the committed outcome is one - // the payer signed. Served only where publishing is switched on (see payer-outcome.js). - ...(e.payerOutcome && payerOutcomeCfg.publish ? { payerOutcome: e.payerOutcome } : {}), - })), - // Payments a reorganization removed (ADR 0012): read again once final and no longer on the chain, so they leave - // the evidence from `removedAt`. A root made before that still holds them. Absent when there are none. - ...removedBody(didHash), - }); + }; +} + +/** + * One payment as /evidence lists it, the leaf at `i` of `tree`. + * + * The leaf is derivable from the payment, so a verifier rebuilds it rather than + * trusting the one served here; it is included to make that comparison easy. + */ +function evidenceEntry(e, i, tree) { + return { + ...(e.chainId != null ? { chainId: e.chainId } : {}), + txHash: e.txHash ?? null, + payer: e.payer, + amount: e.amount, + settledAt: merkle.settledSeconds(e), + success: e.success, + // Who the outcome is the word of (docs/adr/0006-payer-signed-outcomes.md): the payer, by a + // signature anyone can check, or a credential holder. null when there is none. + outcomeSource: outcomeSource(e), + // A payer's "not delivered" (ADR 0011): when it counts from, and the refund that cured it if one did. + // `success` above is how it reads in this evidence, null until it counts. Not part of the leaf. + ...(e.payerOutcome?.outcome === negativeHold.NOT_DELIVERED ? { + payerNegative: { + countsFrom: Number.isFinite(negativeHold.countsFromMs(e, negativeHoldCfg)) + ? new Date(negativeHold.countsFromMs(e, negativeHoldCfg)).toISOString() : null, + settled: e.refundCheck?.final === true, + refundTx: e.refundCheck?.txHash ?? null, + }, + } : {}), + leaf: tree.leaves[i], + proof: merkle.proofFor(tree, i), + // Not part of the leaf and not verified here. See committedFields and unverifiedFields above. + ...(e.packetId ? { counterauditPacketId: e.packetId } : {}), + // The payer's signed payload, so a verifier can check that the committed outcome is one + // the payer signed. Served only where publishing is switched on (see payer-outcome.js). + ...(e.payerOutcome && payerOutcomeCfg.publish ? { payerOutcome: e.payerOutcome } : {}), + }; +} + +// Payments serialized between pauses while a body is built: a few milliseconds of work at a time. +const EVIDENCE_SLICE = 256; +// The most payments one page of /evidence lists (?offset=&limit=). +const EVIDENCE_PAGE_MAX = 1000; + +/** + * The /evidence body for an agent, as a JSON string. + * + * Built once per (agent, payment set). Everything here is a pure function of `events`, + * which is why it caches cleanly: the tree, every proof and the serialization are + * O(E log E) work that does not change until a payment is credited or pruned. + * + * Built a slice at a time with other work let in between, where it was one JSON.stringify: 2.95 s with the event loop + * held for an agent at the cap of 25,000 payments, and /health waited 2.7 s behind one (performance review, + * 15 November). The bytes are the ones JSON.stringify gave for the whole object. + */ +async function buildEvidenceBody(didHash, events) { + // Payments a reorganization removed (ADR 0012): read again once final and no longer on the chain, so they leave + // the evidence from `removedAt`. A root made before that still holds them. Absent when there are none. + const tail = JSON.stringify(removedBody(didHash)); + const tree = await merkle.treeOf(events); + const head = JSON.stringify(evidenceHead(didHash, events, tree.root)); + const parts = [head.slice(0, -1), ',"evidence":[']; + for (let i = 0; i < events.length; i += EVIDENCE_SLICE) { + if (i > 0) await new Promise((resolve) => setImmediate(resolve)); + const slice = []; + for (let j = i; j < Math.min(events.length, i + EVIDENCE_SLICE); j += 1) slice.push(JSON.stringify(evidenceEntry(events[j], j, tree))); + parts.push((i > 0 ? ',' : '') + slice.join(',')); + } + parts.push(']', tail === '{}' ? '' : `,${tail.slice(1, -1)}`, '}'); + return parts.join(''); +} + +/** Payments `offset` to `offset + limit` of the body, with what the body says before them; `next` is where the next page starts. */ +async function evidencePageBody(didHash, events, offset, limit) { + const removed = removedBody(didHash); + const tree = await merkle.treeOf(events); + const end = Math.min(events.length, offset + limit); + const evidence = []; + for (let i = offset; i < end; i += 1) evidence.push(evidenceEntry(events[i], i, tree)); + return JSON.stringify({ ...evidenceHead(didHash, events, tree.root), offset, next: end < events.length ? end : null, evidence, ...removed }); } function removedBody(didHash) { @@ -3526,6 +3598,38 @@ async function handleRequest(req, res) { ...heldOutcomeReply(didHash, body.payment.txHash, signedOutcome, settledOn), }); } + // A settlement already credited to this agent, offered again with no outcome to add, is refused before any + // chain read, as the 409 below refuses it after six. Anyone can copy one from /evidence. Both paths credit + // every qualifying sender of a transaction together (planCredits, verifyPayment), so one live credit means + // the other legs were credited or refused with it. + const offeredTx = typeof body.payment?.txHash === 'string' && TX_HASH.test(body.payment.txHash) + ? body.payment.txHash.toLowerCase() : null; + if (offeredTx && !signedOutcome && outcome === null && payments.required(paymentCfg) + && getPaymentEvents(didHash).some((e) => e.chainId == null && e.txHash === offeredTx && e.removedAt == null)) { + metrics.inc('attestRejectedPayment'); + return json(res, 409, { error: 'this settlement has already been credited', code: 'replayed' }); + } + // A signature that recovers to someone other than the payer, from a payer with no code to vouch for it, is + // refused for one read where the checks below take nine. An address with code is asked below, at the block + // its answer is pinned to (verifySignature), as before. + if (offeredTx && signedOutcome && payments.required(paymentCfg)) { + const signedMsg = payerOutcome.message(didHash, offeredTx, signedOutcome); + if (!payerOutcome.recover(outcomeDomain(settledOn), signedMsg, signedOutcome.signature).valid) { + let code; + try { + code = await (pc ? paymentChains.boundedProvider(paymentChainProviders.get(pc.chainId)) : chain.getProvider()) + .getCode(signedOutcome.payer); + } catch (err) { + metrics.inc('paymentRpcErrors'); + console.warn(`[oracle] /attest code read failed for ${didHash}: ${pc ? paymentChains.rpcError(err, pc.rpcUrl).message : err.message}`); + return json(res, 502, { error: 'could not read the chain', code: 'rpc_error' }); + } + if (!code || code === '0x') { + metrics.inc('attestRejectedOther'); + return json(res, 400, { error: 'the signature is not the payer\'s for this outcome', code: 'bad_signature' }); + } + } + } // Only needed when payments are off; otherwise it comes from the transfer. if (!attester && !payments.required(paymentCfg)) { metrics.inc('attestRejectedOther'); @@ -3734,12 +3838,14 @@ async function handleRequest(req, res) { // and a restart before then loses it: an attested outcome is on no chain to be found again. Answering 200 // would tell the submitter it counted for good. 202 says it was taken and is not yet safe, and /health is // already unhealthy for the same reason. The submitter must not resend: the payment is credited here. - const durable = persistState(); + // Shared with the writes that arrive with it (saveSoon); the tally is this one's, taken before the wait. + const tally = { ...att }; + const durable = await saveSoon(); metrics.inc(durable ? 'attestAccepted' : 'attestAcceptedNotDurable'); return json(res, durable ? 200 : 202, { didHash, attester, - ...att, + ...tally, ...(durable ? {} : { durable: false }), ...(credited ? { payment: { txHash: credited.txHash, amount: credited.amount.toString() }, outcomeRecorded } @@ -3863,7 +3969,27 @@ async function handleRequest(req, res) { } catch { /* this host's clock */ } } const events = evidenceAt(didHash, paymentChains.cutFor(nowMs, paymentChainsCfg)); - const body = evidenceCache.get(didHash, events, () => buildEvidenceBody(didHash, events)); + let body; + if (url.searchParams.has('offset') || url.searchParams.has('limit')) { + // A page of the body, for a verifier that would rather not take one of 40 MB. Asked for by name, so a caller + // that asks for neither still gets the whole body, as every client written before pages does. + const whole = (name, fallback) => { + const raw = url.searchParams.get(name); + return raw === null ? fallback : /^\d{1,9}$/.test(raw) ? Number(raw) : NaN; + }; + const offset = whole('offset', 0); + const limit = whole('limit', EVIDENCE_PAGE_MAX); + if (!Number.isInteger(offset) || !Number.isInteger(limit) || limit < 1 || limit > EVIDENCE_PAGE_MAX) { + return json(res, 400, { error: `offset must be a whole number, and limit one from 1 to ${EVIDENCE_PAGE_MAX}` }); + } + body = await evidencePageBody(didHash, events, offset, limit); + } else if (events.length === 0) { + // Not cached: it costs nothing to build, and one stored for every didHash anyone asked about pushed out the + // bodies worth keeping. + body = await buildEvidenceBody(didHash, events); + } else { + body = await evidenceCache.getAsync(didHash, events, () => buildEvidenceBody(didHash, events)); + } // Same bytes a miss would have produced, so a hit is indistinguishable to the // caller. json() is not used because it would re-serialize what is already a string. diff --git a/oracle/keeper.js b/oracle/keeper.js index 9cf55b0..8b07edc 100644 --- a/oracle/keeper.js +++ b/oracle/keeper.js @@ -37,6 +37,7 @@ const { webhookPayload, trackPollOutcome } = require('./watcher-policy'); const { redact } = require('./event-index'); const { wordsOf } = require('./log-spans'); const { readWithBackoff } = require('./read-retry'); +const { rpcProvider } = require('./rpc-provider'); const STAKING_ABI = [ 'event SlashInitiated(bytes32 indexed didHash, address indexed reporter, address indexed victim, uint256 initiatedAt)', @@ -406,7 +407,7 @@ async function main() { console.error(`[keeper] ${err.message}`); process.exit(1); } - const provider = new ethers.JsonRpcProvider(cfg.rpcUrl); + const provider = rpcProvider(cfg.rpcUrl); const staking = new ethers.Contract(cfg.staking, STAKING_ABI, provider); const wallet = cfg.privateKey ? new ethers.Wallet(cfg.privateKey, provider) : null; // Each read through the backoff the oracle uses: a throttle says nothing about a slash. A real diff --git a/oracle/merkle.js b/oracle/merkle.js index 1136f15..77e7b96 100644 --- a/oracle/merkle.js +++ b/oracle/merkle.js @@ -183,14 +183,35 @@ function hashPair(a, b) { } /** - * Builds the tree over `events` and returns { root, leaves, layers }. + * hashPair through `pairs`, a Map shared by trees over nearly the same leaves, when one is given. Trees over two + * cut-offs of one list hold every pair left of where the lists differ, so a checker trying thirty of them hashes + * those pairs once (evidence-as-of.js). + */ +function pairOf(a, b, pairs) { + if (!pairs) return hashPair(a, b); + const key = a + b; + let hash = pairs.get(key); + if (hash === undefined) { + hash = hashPair(a, b); + pairs.set(key, hash); + } + return hash; +} + +/** + * The work of building a tree, one hash a step, returning { root, leaves, layers }. buildTree runs it straight + * through; treeOf lets other work run between steps. * * An odd node is promoted rather than duplicated. Duplicating it is the classic * malleability bug: a tree can then be extended with the repeated leaf and still * produce the same root. */ -function buildTree(events) { - const leaves = events.map(leafFor); +function* growTree(events, pairs) { + const leaves = []; + for (const event of events) { + leaves.push(leafFor(event)); + yield; + } if (leaves.length === 0) return { root: ethers.ZeroHash, leaves: [], layers: [] }; const layers = [leaves.slice()]; @@ -198,7 +219,8 @@ function buildTree(events) { while (level.length > 1) { const next = []; for (let i = 0; i < level.length; i += 2) { - next.push(i + 1 < level.length ? hashPair(level[i], level[i + 1]) : level[i]); + next.push(i + 1 < level.length ? pairOf(level[i], level[i + 1], pairs) : level[i]); + yield; } layers.push(next); level = next; @@ -206,17 +228,57 @@ function buildTree(events) { return { root: level[0], leaves, layers }; } -function rootFor(events) { +/** Builds the tree over `events` and returns { root, leaves, layers }. `pairs`: see pairOf. */ +function buildTree(events, pairs = null) { + const steps = growTree(events, pairs); + for (;;) { + const step = steps.next(); + if (step.done) return step.value; + } +} + +const frozenThrough = (events) => Array.isArray(events) && Object.isFrozen(events) && events.every((e) => Object.isFrozen(e)); + +function rootFor(events, pairs = null) { const hit = rootCache.get(events); if (hit !== undefined) return hit; - const root = buildTree(events).root; + const root = buildTree(events, pairs).root; // Cached only when nothing in it can change: the array and every event are frozen. - if (Array.isArray(events) && Object.isFrozen(events) && events.every((e) => Object.isFrozen(e))) { - rootCache.set(events, root); - } + if (frozenThrough(events)) rootCache.set(events, root); return root; } +// Trees treeOf built, by events array. Held only while the array is: the store replaces an agent's array on every +// change, so a tree goes with the evidence it was built over. +const treeCache = new WeakMap(); +const TREE_STEPS_BETWEEN_PAUSES = 2048; +const pause = () => new Promise((resolve) => setImmediate(resolve)); + +/** + * buildTree for a caller that must not hold the event loop while it hashes: /evidence, which took 1.4 s for an agent + * at the cap of 25,000 payments. Other work runs every TREE_STEPS_BETWEEN_PAUSES hashes. The tree is kept for a + * frozen array, as rootFor keeps the root, so the next request over the same evidence builds nothing. + */ +async function treeOf(events) { + const hit = treeCache.get(events); + if (hit !== undefined) return hit; + const steps = growTree(events, null); + let tree; + for (let n = 1; ; n += 1) { + const step = steps.next(); + if (step.done) { + tree = step.value; + break; + } + if (n % TREE_STEPS_BETWEEN_PAUSES === 0) await pause(); + } + if (frozenThrough(events)) { + treeCache.set(events, tree); + rootCache.set(events, tree.root); + } + return tree; +} + /// The sibling path for the leaf at `index`, in the order MerkleProof.verify expects. function proofFor(tree, index) { const proof = []; @@ -238,5 +300,5 @@ function verifyProof(leaf, proof, root) { } module.exports = { - leafFor, leafByAbi, buildTree, rootFor, proofFor, verifyProof, hashPair, settledSeconds, outcomeCode, + leafFor, leafByAbi, buildTree, rootFor, treeOf, proofFor, verifyProof, hashPair, settledSeconds, outcomeCode, }; diff --git a/oracle/merkle.test.js b/oracle/merkle.test.js index ea98bc8..22419ed 100644 --- a/oracle/merkle.test.js +++ b/oracle/merkle.test.js @@ -383,3 +383,19 @@ test('rootFor: frozen events inside an array that is not frozen are not cached a events[1] = Object.freeze(ev(9)); assert.notEqual(rootFor(events), before); }); + +// /evidence built its tree with the event loop held: 1.4 s for an agent at the cap of 25,000 payments. +test('treeOf builds the tree buildTree does, lets other work run while it hashes, and keeps it for a frozen list', async () => { + const events = Object.freeze(Array.from({ length: 3000 }, (_, i) => Object.freeze({ + txHash: '0x' + (i + 1).toString(16).padStart(64, '0'), payer: '0x' + '11'.repeat(20), amount: '1000', ts: 1_700_000_000_000 + i, success: true, + }))); + const merkle = require('./merkle'); + let ran = false; + const building = merkle.treeOf(events); + setImmediate(() => { ran = true; }); + const tree = await building; + assert.equal(ran, true, 'something else ran before it finished'); + assert.deepEqual(tree, buildTree(events)); + assert.equal(await merkle.treeOf(events), tree, 'kept, not built again'); + assert.equal(rootFor(events), tree.root); +}); diff --git a/oracle/metrics.js b/oracle/metrics.js index 2942d0e..40dd139 100644 --- a/oracle/metrics.js +++ b/oracle/metrics.js @@ -43,6 +43,9 @@ const counters = { // A flag, a resolve or a link that was taken and held in memory, but the state file could not be written. The // reply says so (202, durable: false). Worth alerting on beside /health, like attestAcceptedNotDurable. writesNotDurable: 0, + // Saves of the whole state file. Each one serializes and fsyncs everything, so a rate that follows /attest + // traffic one for one means writes are not sharing saves. + stateSaves: 0, scoreRpcErrors: 0, scoreErrors: 0, // /score previews answered 503 because the payers' standings were not read yet. A steady count means requests @@ -300,6 +303,11 @@ function toPrometheusText(extra = {}) { lines.push('# TYPE sigvara_oracle_score_standings_pending_total counter'); lines.push(`sigvara_oracle_score_standings_pending_total ${counters.scoreStandingsPending}`); + lines.push(''); + lines.push('# HELP sigvara_oracle_state_saves_total Saves of the whole state file'); + lines.push('# TYPE sigvara_oracle_state_saves_total counter'); + lines.push(`sigvara_oracle_state_saves_total ${counters.stateSaves}`); + lines.push(''); lines.push('# HELP sigvara_oracle_badge_requests_total Badge SVG requests'); lines.push('# TYPE sigvara_oracle_badge_requests_total counter'); diff --git a/oracle/payer-outcome.js b/oracle/payer-outcome.js index 81e41b5..007677a 100644 --- a/oracle/payer-outcome.js +++ b/oracle/payer-outcome.js @@ -504,6 +504,7 @@ module.exports = { timingProblem, checkedBlockProblem, importLateProblem, + recover, verifySignature, verifyImported, RpcError, diff --git a/oracle/payer-outcome.test.js b/oracle/payer-outcome.test.js index eda72d2..daecd5f 100644 --- a/oracle/payer-outcome.test.js +++ b/oracle/payer-outcome.test.js @@ -544,6 +544,22 @@ describe('POST /attest with a payer-signed outcome', () => { } }); + // A forged signature used to cost the payment's reads, then the head, a block and the payer's code, before its 400. + test('a signature that recovers to someone else, from a payer with no code, is refused before the payment is read', async () => { + const payer = ethers.Wallet.createRandom(); + const { agent, txHash } = paid(payer); + const receiptReads = () => oracle.calls().filter((c) => c.name === 'receipt' && c.args[0] === txHash.toLowerCase()).length; + const forged = await signOutcome(agent.didHash, txHash, payer, { signer: ethers.Wallet.createRandom() }); + const r = await post(oracle, agent.didHash, txHash, forged); + assert.equal(r.status, 400, r.text); + assert.equal(r.body.code, 'bad_signature'); + assert.equal(receiptReads(), 0, 'refused without reading the payment'); + // The payer's own signature still has the whole check. + const own = await post(oracle, agent.didHash, txHash, await signOutcome(agent.didHash, txHash, payer)); + assert.equal(own.status, 200, own.text); + assert.equal(receiptReads(), 1); + }); + test('refuses a signature dated before settlement or after its deadline', async () => { const payer = ethers.Wallet.createRandom(); const { agent, txHash } = paid(payer); diff --git a/oracle/payment-chain-operators.test.js b/oracle/payment-chain-operators.test.js index 31db786..0d99cf2 100644 --- a/oracle/payment-chain-operators.test.js +++ b/oracle/payment-chain-operators.test.js @@ -180,9 +180,9 @@ test('two operators, one scanning Base before it reads the handover and one afte }; const held = pc.eventsAt(Object.freeze(checker.credited), atProposalMs, cfg); const early = merkle.rootFor(pc.eventsAt(Object.freeze(primary.credited), HANDOVER_MS - 5 * 60_000, cfg, namedAt)); - assert.equal(evidenceAsOf(held, atProposalMs, early, options).matchedRoot, true, 'a cut-off before the handover'); + assert.equal((await evidenceAsOf(held, atProposalMs, early, options)).matchedRoot, true, 'a cut-off before the handover'); const late = merkle.rootFor(pc.eventsAt(Object.freeze(primary.credited), HANDOVER_MS + 5 * 60_000, cfg, namedAt)); - const got = evidenceAsOf(held, atProposalMs, late, options); + const got = await evidenceAsOf(held, atProposalMs, late, options); assert.equal(got.matchedRoot, true, 'a cut-off after it'); assert.equal(got.events.length, 1); }); diff --git a/oracle/payment-chain-outcomes.test.js b/oracle/payment-chain-outcomes.test.js index a911f94..febf0a7 100644 --- a/oracle/payment-chain-outcomes.test.js +++ b/oracle/payment-chain-outcomes.test.js @@ -376,7 +376,9 @@ describe('POST /attest for a payment on a payment chain', () => { assert.equal(early.status, 503, early.text); assert.equal(early.body.code, 'unconfirmed'); - const unknown = await post(oracle, agent.didHash, ethers.id('not mined'), signed); + // Signed for the hash it names: a signature for another payment is refused at once (400, bad_signature). + const notMined = ethers.id('not mined'); + const unknown = await post(oracle, agent.didHash, notMined, await signOutcome(agent.didHash, notMined, payer)); assert.equal(unknown.status, 503, unknown.text); assert.equal(unknown.body.code, 'not_found'); diff --git a/oracle/payment-chains.test.js b/oracle/payment-chains.test.js index 434212f..5f58d6d 100644 --- a/oracle/payment-chains.test.js +++ b/oracle/payment-chains.test.js @@ -727,7 +727,7 @@ test('with no payment chain the order is fixed too, and every stored payment sti } }); -test('when no cut-off matches the root, a checker scores what counted by the newest one a proposer could use', () => { +test('when no cut-off matches the root, a checker scores what counted by the newest one a proposer could use', async () => { const { evidenceAsOf } = require('./evidence-as-of'); const cfg = config(); const P = NOW + 25_000; @@ -735,7 +735,7 @@ test('when no cut-off matches the root, a checker scores what counted by the new const pastTheCut = arcEvent(NOW - 30_000); // after the newest cut-off, NOW - 60 s, and before P const options = { eligibleAt: (e) => pc.eligibleAtMs(e, cfg), cuts: pc.auditCuts(P, cfg) }; assert.equal(options.cuts[0], pc.cutFor(P, cfg)); - const got = evidenceAsOf(pc.eventsAt(Object.freeze([old, pastTheCut]), P, cfg), P, `0x${'ab'.repeat(32)}`, options); + const got = await evidenceAsOf(pc.eventsAt(Object.freeze([old, pastTheCut]), P, cfg), P, `0x${'ab'.repeat(32)}`, options); assert.equal(got.matchedRoot, false); assert.deepEqual(got.events, [old]); }); @@ -843,7 +843,7 @@ test('an RPC URL is https, or http only to this host or a private network', () = } }); -test('a score is cut at a whole minute, and a checker finds that cut from the proposal alone', () => { +test('a score is cut at a whole minute, and a checker finds that cut from the proposal alone', async () => { const { evidenceAsOf } = require('./evidence-as-of'); const cfg = config(); const C = NOW + 25_000; // the primary's clock when it scored this agent @@ -860,15 +860,15 @@ test('a score is cut at a whole minute, and a checker finds that cut from the pr const root = merkle.rootFor(proposed); const atP = pc.eventsAt(stored, P, cfg); assert.equal(atP.length, 3); - assert.equal(evidenceAsOf(atP, P, root).matchedRoot, false, 'one cut at the proposal time misses it'); + assert.equal((await evidenceAsOf(atP, P, root)).matchedRoot, false, 'one cut at the proposal time misses it'); const options = { eligibleAt: (e) => pc.eligibleAtMs(e, cfg), cuts: pc.auditCuts(P, cfg) }; assert.equal(options.cuts.length, 29, 'every whole minute in the half hour, from the cut-off of a score made at P'); - const found = evidenceAsOf(atP, P, root, options); + const found = await evidenceAsOf(atP, P, root, options); assert.equal(found.matchedRoot, true); assert.deepEqual(found.events, [old]); // Evidence cut more than half an hour before the proposal is not believed. const lagged = merkle.rootFor(pc.eventsAt(Object.freeze([old, arcEvent(T - 40 * 60_000)]), T - 35 * 60_000, cfg)); - assert.equal(evidenceAsOf(pc.eventsAt(Object.freeze([old, arcEvent(T - 40 * 60_000)]), P, cfg), P, lagged, options).matchedRoot, false); + assert.equal((await evidenceAsOf(pc.eventsAt(Object.freeze([old, arcEvent(T - 40 * 60_000)]), P, cfg), P, lagged, options)).matchedRoot, false); // And the browser finds it too, from the proposal's time and what /evidence serves. const served = (e) => ({ ...(e.chainId != null ? { chainId: e.chainId } : {}), txHash: e.txHash, payer: e.payer, amount: e.amount, settledAt: merkle.settledSeconds(e), success: e.success }); const evidenceCut = { quantumSeconds: 60, windowSeconds: 1800 }; diff --git a/oracle/publisher.js b/oracle/publisher.js index 6fc58ce..96b4968 100644 --- a/oracle/publisher.js +++ b/oracle/publisher.js @@ -28,6 +28,7 @@ const { ethers } = require('ethers'); const policy = require('./publisher-policy'); const { createLinkScanner } = require('./link-scan'); const signerAuth = require('./signer-auth'); +const { rpcProvider } = require('./rpc-provider'); const REPUTATION_EVENTS = [ 'event ReputationUpdated(bytes32 indexed didHash, uint8 totalScore, uint256 timestamp, bytes32 evidenceRoot)', @@ -445,7 +446,7 @@ async function main() { } const chain = require('./chain'); const external = require('./external'); - const provider = new ethers.JsonRpcProvider(cfg.rpcUrl); + const provider = rpcProvider(cfg.rpcUrl); // The chain module's reads, with no key: the provider stands in for its wallet. chain.init({ rpcUrl: cfg.rpcUrl, identityAddress: cfg.identityAddress, reputationAddress: cfg.reputation }, { provider, wallet: provider }); external.init({ externalRpc: cfg.externalRpc, externalIdentity: cfg.identityRegistry, externalReputation: cfg.feedbackRegistry }); diff --git a/oracle/response-cache.js b/oracle/response-cache.js index e240605..b6ef69e 100644 --- a/oracle/response-cache.js +++ b/oracle/response-cache.js @@ -52,40 +52,64 @@ function createResponseCache({ maxBytes }) { } } + // The stored body for this revision, moved to the back of the LRU order, or undefined. + function take(key, revision) { + const hit = entries.get(key); + if (!hit || hit.revision !== revision) return undefined; + entries.delete(key); + entries.set(key, hit); + stats.hits++; + return hit.body; + } + + function store(key, revision, body) { + const bytes = Buffer.byteLength(body); + // A stale entry for this key is replaced, not left behind. + const old = entries.get(key); + if (old) { + totalBytes -= old.bytes; + entries.delete(key); + } + // Serve, but do not store, anything that cannot coexist with a useful cache. + if (bytes > maxBytes) return; + entries.set(key, { revision, body, bytes }); + totalBytes += bytes; + evictTo(maxBytes); + } + + // Builds in flight, by key: { revision, body: Promise }. + const building = new Map(); + return { /** - * Serialized body for `key`, built by `build()` on a miss. + * Serialized body for `key`, built by `build()` on a miss. `build` returns the finished string or a promise of + * it, and is called only when needed, so an expensive build is skipped entirely on a hit. * - * `build` must return the finished string. It is called synchronously and only when - * needed, so an expensive build is skipped entirely on a hit. + * The body is built without holding the event loop (index.js), so a request can arrive while the same body is + * being built. It waits for that build instead of starting its own, which would hold a second copy until sent. */ - get(key, revision, build) { - const hit = entries.get(key); - if (hit && hit.revision === revision) { - // Re-insert to move it to the back of the LRU order. - entries.delete(key); - entries.set(key, hit); + getAsync(key, revision, build) { + const hit = take(key, revision); + if (hit !== undefined) return Promise.resolve(hit); + const pending = building.get(key); + if (pending && pending.revision === revision) { stats.hits++; - return hit.body; + return pending.body; } stats.misses++; - - const body = build(); - const bytes = Buffer.byteLength(body); - - // A stale entry for this key is replaced, not left behind. - if (hit) { - totalBytes -= hit.bytes; - entries.delete(key); - } - - // Serve, but do not store, anything that cannot coexist with a useful cache. - if (bytes > maxBytes) return body; - - entries.set(key, { revision, body, bytes }); - totalBytes += bytes; - evictTo(maxBytes); - return body; + const entry = { revision, body: null }; + entry.body = Promise.resolve() + .then(build) + .then((body) => { + // Not when a build for a later revision has started since: that one is the body to keep. + if (building.get(key) === entry) store(key, revision, body); + return body; + }) + .finally(() => { + if (building.get(key) === entry) building.delete(key); + }); + building.set(key, entry); + return entry.body; }, stats() { diff --git a/oracle/response-cache.test.js b/oracle/response-cache.test.js index f8385ce..c2f8f9c 100644 --- a/oracle/response-cache.test.js +++ b/oracle/response-cache.test.js @@ -4,20 +4,20 @@ const { test } = require('node:test'); const assert = require('node:assert/strict'); const { createResponseCache } = require('./response-cache'); -test('createResponseCache: builds on a miss, reuses on a hit', () => { +test('createResponseCache: builds on a miss, reuses on a hit', async () => { const cache = createResponseCache({ maxBytes: 1024 }); const rev = ['a']; let builds = 0; const build = () => { builds++; return '{"n":1}'; }; - assert.equal(cache.get('k', rev, build), '{"n":1}'); - assert.equal(cache.get('k', rev, build), '{"n":1}'); - assert.equal(cache.get('k', rev, build), '{"n":1}'); + assert.equal(await cache.getAsync('k', rev, build), '{"n":1}'); + assert.equal(await cache.getAsync('k', rev, build), '{"n":1}'); + assert.equal(await cache.getAsync('k', rev, build), '{"n":1}'); assert.equal(builds, 1, 'the expensive build should run once'); assert.equal(cache.stats().hits, 2); }); -test('createResponseCache: a new revision rebuilds', () => { +test('createResponseCache: a new revision rebuilds', async () => { // The whole invalidation story. For /evidence the revision is the payment-event // array, which the store replaces on every credit and every prune, so a changed // payment set is a changed identity and nothing has to call invalidate(). @@ -25,94 +25,127 @@ test('createResponseCache: a new revision rebuilds', () => { const before = ['p1']; const after = ['p1', 'p2']; - assert.equal(cache.get('agent', before, () => 'one'), 'one'); - assert.equal(cache.get('agent', after, () => 'two'), 'two'); + assert.equal(await cache.getAsync('agent', before, () => 'one'), 'one'); + assert.equal(await cache.getAsync('agent', after, () => 'two'), 'two'); // And it does not fall back to the old entry afterwards. - assert.equal(cache.get('agent', after, () => 'three'), 'two'); + assert.equal(await cache.getAsync('agent', after, () => 'three'), 'two'); }); -test('createResponseCache: identity, not equality', () => { +test('createResponseCache: identity, not equality', async () => { // Two arrays with identical contents are different revisions. That is the safe // direction: a false miss costs a rebuild, a false hit serves stale evidence. const cache = createResponseCache({ maxBytes: 1024 }); let builds = 0; const build = () => { builds++; return 'body'; }; - cache.get('k', ['same'], build); - cache.get('k', ['same'], build); + await cache.getAsync('k', ['same'], build); + await cache.getAsync('k', ['same'], build); assert.equal(builds, 2); }); -test('createResponseCache: replacing an entry does not leak its bytes', () => { +test('createResponseCache: replacing an entry does not leak its bytes', async () => { const cache = createResponseCache({ maxBytes: 1024 }); - cache.get('k', ['r1'], () => 'x'.repeat(100)); + await cache.getAsync('k', ['r1'], () => 'x'.repeat(100)); const afterFirst = cache.stats().bytes; - cache.get('k', ['r2'], () => 'y'.repeat(100)); + await cache.getAsync('k', ['r2'], () => 'y'.repeat(100)); assert.equal(cache.stats().entries, 1); assert.equal(cache.stats().bytes, afterFirst, 'the stale entry’s bytes are reclaimed'); }); -test('createResponseCache: evicts least-recently-used to stay under the byte cap', () => { +test('createResponseCache: evicts least-recently-used to stay under the byte cap', async () => { const cache = createResponseCache({ maxBytes: 300 }); // One shared revision object: a fresh ['r'] per call is a different identity and // every lookup would miss, which is what the test above pins down. const rev = ['r']; - cache.get('a', rev, () => 'a'.repeat(100)); - cache.get('b', rev, () => 'b'.repeat(100)); - cache.get('c', rev, () => 'c'.repeat(100)); + await cache.getAsync('a', rev, () => 'a'.repeat(100)); + await cache.getAsync('b', rev, () => 'b'.repeat(100)); + await cache.getAsync('c', rev, () => 'c'.repeat(100)); assert.equal(cache.stats().entries, 3); // Touch 'a' so 'b' becomes the oldest, then overflow by one entry. - cache.get('a', rev, () => { throw new Error('should be a hit'); }); - cache.get('d', rev, () => 'd'.repeat(100)); + await cache.getAsync('a', rev, () => { throw new Error('should be a hit'); }); + await cache.getAsync('d', rev, () => 'd'.repeat(100)); assert.ok(cache.stats().bytes <= 300, 'stays under the cap'); // 'b' was least recently used, so it went first; 'a' survived because it was touched. let rebuiltA = false; - cache.get('a', rev, () => { rebuiltA = true; return 'a'.repeat(100); }); + await cache.getAsync('a', rev, () => { rebuiltA = true; return 'a'.repeat(100); }); assert.equal(rebuiltA, false, 'the recently used entry survived'); // And 'b' really is the one that went. let rebuiltB = false; - cache.get('b', rev, () => { rebuiltB = true; return 'b'.repeat(100); }); + await cache.getAsync('b', rev, () => { rebuiltB = true; return 'b'.repeat(100); }); assert.equal(rebuiltB, true, 'the least recently used entry was evicted'); }); -test('createResponseCache: an oversized body is served but not stored', () => { +test('createResponseCache: an oversized body is served but not stored', async () => { // One agent with an enormous payment history must not evict the entire cache to // hold a response nothing else can coexist with. const cache = createResponseCache({ maxBytes: 100 }); const rev = ['r']; - cache.get('small', rev, () => 's'.repeat(50)); + await cache.getAsync('small', rev, () => 's'.repeat(50)); const huge = 'h'.repeat(500); - assert.equal(cache.get('huge', rev, () => huge), huge, 'still served'); + assert.equal(await cache.getAsync('huge', rev, () => huge), huge, 'still served'); assert.equal(cache.stats().entries, 1, 'not stored'); let rebuiltSmall = false; - cache.get('small', rev, () => { rebuiltSmall = true; return 's'.repeat(50); }); + await cache.getAsync('small', rev, () => { rebuiltSmall = true; return 's'.repeat(50); }); assert.equal(rebuiltSmall, false, 'the small entry was not evicted for it'); }); -test('createResponseCache: keys do not collide', () => { +test('createResponseCache: keys do not collide', async () => { const cache = createResponseCache({ maxBytes: 1024 }); const rev = ['r']; - assert.equal(cache.get('agent-1', rev, () => 'one'), 'one'); - assert.equal(cache.get('agent-2', rev, () => 'two'), 'two'); - assert.equal(cache.get('agent-1', rev, () => 'nope'), 'one'); + assert.equal(await cache.getAsync('agent-1', rev, () => 'one'), 'one'); + assert.equal(await cache.getAsync('agent-2', rev, () => 'two'), 'two'); + assert.equal(await cache.getAsync('agent-1', rev, () => 'nope'), 'one'); }); -test('createResponseCache: rejects a cap that would not bound anything', () => { +test('createResponseCache: rejects a cap that would not bound anything', async () => { for (const bad of [0, -1, 1.5, NaN, undefined]) { assert.throws(() => createResponseCache({ maxBytes: bad }), TypeError); } }); -test('createResponseCache: measures bytes, not characters', () => { +test('createResponseCache: measures bytes, not characters', async () => { // Buffer.byteLength, not .length: a multi-byte body would otherwise be undercounted // and the cap would not hold. const cache = createResponseCache({ maxBytes: 1024 }); - cache.get('k', ['r'], () => 'é'.repeat(10)); // 10 chars, 20 bytes + await cache.getAsync('k', ['r'], () => 'é'.repeat(10)); // 10 chars, 20 bytes assert.equal(cache.stats().bytes, 20); }); + +// /evidence builds its body without holding the event loop, so two requests can now arrive while one is building. +test('getAsync: requests that arrive while a body is building wait for that build', async () => { + const cache = createResponseCache({ maxBytes: 1024 }); + const rev = ['a']; + let builds = 0; + let finish; + const build = () => { builds++; return new Promise((resolve) => { finish = () => resolve('{"n":1}'); }); }; + const first = cache.getAsync('k', rev, build); + const second = cache.getAsync('k', rev, build); + await new Promise((resolve) => setImmediate(resolve)); + finish(); + assert.deepEqual(await Promise.all([first, second]), ['{"n":1}', '{"n":1}']); + assert.equal(builds, 1); + assert.equal(await cache.getAsync('k', rev, build), '{"n":1}', 'then served from the cache'); + assert.equal(builds, 1); +}); + +test('getAsync: a build that finishes after one for a later revision started is served, not kept', async () => { + const cache = createResponseCache({ maxBytes: 1024 }); + const older = ['a']; + const newer = ['b']; + const finishes = []; + const build = (body) => () => new Promise((resolve) => { finishes.push(() => resolve(body)); }); + const slow = cache.getAsync('k', older, build('{"v":1}')); + const fresh = cache.getAsync('k', newer, build('{"v":2}')); + await new Promise((resolve) => setImmediate(resolve)); + finishes[1](); + assert.equal(await fresh, '{"v":2}'); + finishes[0](); + assert.equal(await slow, '{"v":1}'); + assert.equal(await cache.getAsync('k', newer, () => { throw new Error('built again'); }), '{"v":2}'); +}); diff --git a/oracle/rpc-provider.js b/oracle/rpc-provider.js new file mode 100644 index 0000000..45d7632 --- /dev/null +++ b/oracle/rpc-provider.js @@ -0,0 +1,31 @@ +'use strict'; + +// How the oracle and its services reach a node. On its own, beside read-retry.js, so read-retry.js keeps no +// dependencies. + +const { ethers } = require('ethers'); +const { MAX_BATCH_CALLS } = require('./read-retry'); + +// How long one request may take. The payment chains' figure (payment-chains.RPC_TIMEOUT_MS), which a wide log read +// on Arc and Base answers well inside. ethers' own default is 300 s. +const RPC_TIMEOUT_MS = 30_000; + +/** + * A provider for `url`, built the same way everywhere: + * - each request has a deadline, where ethers' default let a stalled node hold /score, /attest and /link for + * five minutes a read; + * - ethers tries nothing again itself. It tried an HTTP 429 up to 12 times inside each of readWithBackoff's + * attempts, so a throttling node was still being asked after 100 s (performance review, 15 November). + * A throttle is retried by readWithBackoff, and only there; + * - the chain is fixed: `chainId` when it is known, the node's first answer otherwise. Without that ethers asks + * eth_chainId before every wave of reads. Whatever has to know the chain the node serves now asks it directly + * (chain.getChainId, external.refreshChainId), as before. + */ +function rpcProvider(url, chainId = null, { timeoutMs = RPC_TIMEOUT_MS, batchMaxCount = MAX_BATCH_CALLS } = {}) { + const request = new ethers.FetchRequest(url); + request.timeout = timeoutMs; + request.setThrottleParams({ maxAttempts: 1 }); + return new ethers.JsonRpcProvider(request, chainId == null ? undefined : Number(chainId), { staticNetwork: true, batchMaxCount }); +} + +module.exports = { rpcProvider, RPC_TIMEOUT_MS }; diff --git a/oracle/rpc-provider.test.js b/oracle/rpc-provider.test.js new file mode 100644 index 0000000..4139359 --- /dev/null +++ b/oracle/rpc-provider.test.js @@ -0,0 +1,88 @@ +'use strict'; + +// How the oracle reaches a node (rpc-provider.js), against a local JSON-RPC endpoint that counts what it is asked. + +const { test, after } = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('node:http'); +const { ethers } = require('ethers'); +const { rpcProvider } = require('./rpc-provider'); +const chain = require('./chain'); + +const CHAIN_ID = 5042002; +const asked = []; +let mode = 'answer'; +const server = http.createServer((req, res) => { + let body = ''; + req.on('data', (c) => { body += c; }); + req.on('end', () => { + const calls = [].concat(JSON.parse(body)); + for (const c of calls) asked.push(c.method); + if (mode === 'silent') return; // never answers + if (mode === 'throttle') { res.writeHead(429); return res.end('{}'); } + const answer = (c) => ({ + jsonrpc: '2.0', + id: c.id, + result: c.method === 'eth_chainId' ? ethers.toQuantity(CHAIN_ID) : c.method === 'eth_getBalance' ? '0x1' : '0x10', + }); + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify(Array.isArray(JSON.parse(body)) ? calls.map(answer) : answer(calls[0]))); + }); +}); +const url = new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve(`http://127.0.0.1:${server.address().port}`))); +after(() => { server.closeAllConnections(); server.close(); }); + +const count = (method) => asked.filter((m) => m === method).length; +const someone = (n) => `0x${String(n).padStart(40, '0')}`; + +// ethers asked eth_chainId before every wave of reads: 21 to 29% of what the SDK sent (performance review, 15 November). +test('the oracle\'s provider asks no eth_chainId with its reads', async () => { + chain.init({ + rpcUrl: await url, + expectedChainId: CHAIN_ID, + privateKey: ethers.Wallet.createRandom().privateKey, + identityAddress: someone(1), + reputationAddress: someone(2), + }); + const p = chain.getProvider(); + asked.length = 0; + for (let i = 0; i < 3; i += 1) await p.getBalance(someone(10 + i)); + assert.equal(count('eth_getBalance'), 3); + assert.equal(count('eth_chainId'), 0); + // What checks the chain asks the node itself. + assert.equal(await chain.getChainId(), CHAIN_ID); + assert.equal(count('eth_chainId'), 1); +}); + +test('without a chain id, the node is asked once, not once a wave', async () => { + const p = rpcProvider(await url); + asked.length = 0; + for (let i = 0; i < 3; i += 1) await p.getBalance(someone(20 + i)); + assert.equal(count('eth_getBalance'), 3); + assert.equal(count('eth_chainId'), 1); +}); + +// ethers tried an HTTP 429 up to 12 times inside each of readWithBackoff's attempts. +test('a throttled request is not tried again by the provider', async () => { + const p = rpcProvider(await url, CHAIN_ID); + asked.length = 0; + mode = 'throttle'; + try { + await assert.rejects(p.send('eth_blockNumber', []), (err) => err.code === 'SERVER_ERROR' && /429/.test(err.message)); + } finally { + mode = 'answer'; + } + assert.equal(count('eth_blockNumber'), 1); +}); + +test('a request with no answer gives up at its deadline', async () => { + const p = rpcProvider(await url, CHAIN_ID, { timeoutMs: 300 }); + mode = 'silent'; + const started = Date.now(); + try { + await assert.rejects(p.send('eth_blockNumber', []), (err) => err.code === 'TIMEOUT'); + } finally { + mode = 'answer'; + } + assert.ok(Date.now() - started < 5_000); +}); diff --git a/oracle/signer.js b/oracle/signer.js index 9791b86..47f9938 100644 --- a/oracle/signer.js +++ b/oracle/signer.js @@ -31,6 +31,8 @@ const path = require('path'); const http = require('http'); const { ethers } = require('ethers'); const { HEADER, MAX_SKEW_SECONDS, authProblem, readSecret } = require('./signer-auth'); +const { readWithBackoff } = require('./read-retry'); +const { rpcProvider } = require('./rpc-provider'); const FEEDBACK = new ethers.Interface([ 'function giveFeedback(uint256 agentId, int128 value, uint8 valueDecimals, string tag1, string tag2, string endpoint, string feedbackURI, bytes32 feedbackHash)', @@ -308,11 +310,12 @@ async function main(argv = process.argv.slice(2), env = process.env) { const password = fs.readFileSync(privateFile(cfg.passwordFile), 'utf8').trim(); const wallet = await ethers.Wallet.fromEncryptedJson(fs.readFileSync(privateFile(cfg.keystore), 'utf8'), password); const secret = readSecret(privateFile(cfg.secretFile)); - const provider = new ethers.JsonRpcProvider(cfg.rpcUrl); - const chain = async () => { + const provider = rpcProvider(cfg.rpcUrl, cfg.chainId); + // Retried here on a throttle or a timeout: the provider tries nothing again itself (rpc-provider.js). + const chain = () => readWithBackoff('signer chain', async () => { const [pendingNonce, block] = await Promise.all([provider.getTransactionCount(wallet.address, 'pending'), provider.getBlock('latest')]); return { pendingNonce, time: Number(block.timestamp) }; - }; + }); fs.mkdirSync(path.dirname(cfg.stateFile), { recursive: true }); const policy = createPolicy({ ...cfg, address: wallet.address }); const handler = createSigner({ diff --git a/oracle/store.js b/oracle/store.js index 5e14cd6..dc23af4 100644 --- a/oracle/store.js +++ b/oracle/store.js @@ -1167,13 +1167,28 @@ function payerOutcomesSince(since, limit) { if (acceptedOrder[mid].acceptedAt > since) hi = mid; else lo = mid + 1; } + // Each agent's outcomes by when they were taken, indexed once per page for the agents it names. Looking each + // one up with .find made a page cost its size times the agent's events: about 1 s for the last page of 25,000. + const indexes = new Map(); + const outcomeAt = (did, acceptedAt, tx, who) => { + let index = indexes.get(did); + if (!index) { + index = new Map(); + for (const x of paymentEvents.get(did) ?? []) { + if (!x.payerOutcome) continue; + const key = `${x.payerOutcome.acceptedAt}|${x.txHash}|${String(x.payer).toLowerCase()}`; + // The first, as .find gave. + if (!index.has(key)) index.set(key, x); + } + indexes.set(did, index); + } + return index.get(`${acceptedAt}|${tx}|${who}`); + }; const outcomes = []; let i = lo; for (; i < acceptedOrder.length && outcomes.length < limit; i += 1) { const { acceptedAt, did, tx, who } = acceptedOrder[i]; - const e = (paymentEvents.get(did) ?? []).find( - (x) => x.txHash === tx && String(x.payer).toLowerCase() === who && x.payerOutcome?.acceptedAt === acceptedAt, - ); + const e = outcomeAt(did, acceptedAt, tx, who); if (e) { outcomes.push({ didHash: did, diff --git a/oracle/store.test.js b/oracle/store.test.js index 8385b7a..ed7b0f4 100644 --- a/oracle/store.test.js +++ b/oracle/store.test.js @@ -1258,3 +1258,33 @@ test('past the per-agent or total cap the payment that matters least is dropped, ledger: { events: 3, maxTotal: 3, maxPerAgent: 2, dropped: 2, withOutcome: 0, droppedWithOutcome: 0, droppedKeys: 2, agentsWithFloor: 0 }, }); }); + +// ---- GET /outcomes ------------------------------------------------------------------------------------------------ + +// Each entry of a page used to be looked up with .find over its agent's whole list, so the last page of a busy agent +// cost its 500 entries times every event the agent held: about 1 s at 25,000 (performance review, 15 November). +test('a page of outcomes costs its agents\' events once, not once an entry', () => { + const out = inChildWithState(null, ` + process.env.PAYMENT_MAX_EVENTS_PER_AGENT = '25000'; + const s = require('./store'); + const did = '0x' + 'aa'.repeat(32), payer = '0x' + '11'.repeat(20); + const tx = (n) => '0x' + n.toString(16).padStart(64, '0'); + const N = 20000, PAGE = 500; + s.creditPayments(Array.from({ length: N }, (_, n) => ({ didHash: did, txHash: tx(n + 1), amount: '5', payer, ts: 1000 + n }))); + const signed = { outcome: 1, signedAt: 1, deadline: 9, signature: '0x', signer: 'eoa', checkedAtBlock: 40, checkedAtMs: 4000 }; + // The last ones in the list, where a .find has the farthest to go. + for (let n = N - PAGE + 1; n <= N; n += 1) s.recordPayerOutcome(did, tx(n), payer, true, signed, { windowMs: 1 }); + const t = process.hrtime.bigint(); + const page = s.payerOutcomesSince(0, PAGE); + const ms = Number(process.hrtime.bigint() - t) / 1e6; + console.log(JSON.stringify({ + ms, count: page.outcomes.length, more: page.more, + first: Number(page.outcomes[0].txHash), last: Number(page.outcomes.at(-1).txHash), + ordered: page.outcomes.every((o, i) => i === 0 || o.acceptedAt > page.outcomes[i - 1].acceptedAt), + })); + `); + const got = JSON.parse(out.trim().split('\n').at(-1)); + assert.deepEqual({ ...got, ms: undefined }, { ms: undefined, count: 500, more: false, first: 19501, last: 20000, ordered: true }); + // About 1 s with a .find an entry on the machine this was written on, about 10 ms indexed. + assert.ok(got.ms < 200, `a page of 500 took ${got.ms.toFixed(0)} ms`); +}); diff --git a/oracle/test-support/stub-chain.js b/oracle/test-support/stub-chain.js index 7ffd90e..a1a9544 100644 --- a/oracle/test-support/stub-chain.js +++ b/oracle/test-support/stub-chain.js @@ -142,7 +142,10 @@ function linkAt(agentId, block) { } const provider = { - getTransactionReceipt: async (hash) => world().receipts[String(hash).toLowerCase()] ?? null, + getTransactionReceipt: async (hash) => { + record('receipt', String(hash).toLowerCase()); + return world().receipts[String(hash).toLowerCase()] ?? null; + }, getBlockNumber: async () => world().head + moved, // 'finalized' is the head unless world.finalized holds it back (ADR 0012). A numbered block lists its transactions // only where world.blockTxs[n] says what they are: a world holds payments it has no block for, and a block without its diff --git a/oracle/watcher.js b/oracle/watcher.js index 35bdb5a..78e05cb 100644 --- a/oracle/watcher.js +++ b/oracle/watcher.js @@ -39,6 +39,7 @@ const { const { redact } = require('./event-index'); const { wordsOf } = require('./log-spans'); const { readWithBackoff } = require('./read-retry'); +const { rpcProvider } = require('./rpc-provider'); const cfg = { checkerUrl: (process.env.CHECKER_URL || '').replace(/\/+$/, ''), @@ -70,7 +71,7 @@ const ABI = [ 'function challengeWindow() view returns (uint256)', ]; -const provider = new ethers.JsonRpcProvider(cfg.rpcUrl); +const provider = rpcProvider(cfg.rpcUrl); const reputation = new ethers.Contract(cfg.reputationAddress, ABI, provider); // An error as words that can leave this machine. An ethers error's message holds the whole RPC URL diff --git a/oracle/write-spend.test.js b/oracle/write-spend.test.js index bf84c1f..01a6acd 100644 --- a/oracle/write-spend.test.js +++ b/oracle/write-spend.test.js @@ -340,12 +340,38 @@ test('bond watch: lets an agent back for a log naming it, everyone for a new min assert.ok((await refresh(240_000)).error, 'a read that fails is said'); assert.equal(watch.size(), 0, 'and lets everyone back'); - watch.add('0xaa'); + watch.add('0xaa'); // found by the epoch at 240 s changes = { agents: new Set(), minimumChanged: false }; await refresh(300_000); assert.equal(watch.excluded('0xaa'), true); - await refresh(24 * HOUR * 1000); - assert.equal(watch.size(), 0, 'once a day everyone is read again'); + await refresh(24 * HOUR * 1000 + 240_000); + assert.equal(watch.size(), 0, 'a day after an agent was found it is read again'); +}); + +// Every agent left out was read again in one epoch once a day: about 0.47 s each, 78 minutes at 10,000 registrations. +test('bond watch: the daily reads of agents left out are spread over the day\'s epochs, the earliest found first', async () => { + const { createBondWatch } = require('./bond-watch'); + const watch = createBondWatch(); + const refresh = (nowMs) => watch.refresh({ + readHead: async () => 1_000, readChanges: async () => ({ agents: new Set(), minimumChanged: false }), nowMs, + }); + await refresh(0); + const all = Array.from({ length: 240 }, (_, i) => `0x${i.toString(16).padStart(64, '0')}`); + for (const didHash of all) watch.add(didHash); + const firstRead = new Map(); + const perEpoch = []; + for (let h = 1; h <= 24; h += 1) { + await refresh(h * HOUR * 1000); + const back = all.filter((didHash) => !watch.excluded(didHash)); + perEpoch.push(back.length); + // Each is read, found still unbonded, and left out again. + for (const didHash of back) { + if (!firstRead.has(didHash)) firstRead.set(didHash, h); + watch.add(didHash); + } + } + assert.deepEqual(perEpoch, Array(24).fill(10), 'a 24th of them each hourly epoch'); + assert.deepEqual([...firstRead.keys()], all, 'every one within the day, in the order they were found'); }); test('writeDueAt: due at once unless the score on chain is the same one, and then once its last write is the refresh old', () => {