Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -601,10 +601,10 @@ Everything below runs offline, with no deployed contracts, except three fork sui
| Suite | Count | Command |
|---|---|---|
| Contracts: unit, fuzz, E2E, upgrade, deploy-script, handover and 22 invariants | 815 | `forge test` |
| Oracle | 1546 | `cd oracle && node --test` |
| Oracle | 1549 | `cd oracle && node --test` |
| SDK | 392 | `cd packages/sdk && npx vitest run` |

Counts are individual tests, so each of the 22 invariants counts once. The contracts were counted on 15 November 2026, the SDK on 15 November and the oracle on 15 November. Without a fork URL, 803 of the contract tests run and 12 skip. Run after `npm run build`, 383 of the SDK's 392 pass and 9 skip: the 8 integration tests, which need a live deployment, and one check that runs only in CI. Without a build, the 9 built-package tests skip too. Vitest strips types without checking them, so `npm run typecheck` in `packages/sdk` checks the tests against the SDK's types as well as the sources. [`scripts/mutation`](scripts/mutation/README.md) holds the harnesses that measure whether these tests would catch a deliberate break in the contracts or deploy scripts.
Counts are individual tests, so each of the 22 invariants counts once. The contracts were counted on 15 November 2026, the SDK on 15 November and the oracle on 16 November. Without a fork URL, 803 of the contract tests run and 12 skip. Run after `npm run build`, 383 of the SDK's 392 pass and 9 skip: the 8 integration tests, which need a live deployment, and one check that runs only in CI. Without a build, the 9 built-package tests skip too. Vitest strips types without checking them, so `npm run typecheck` in `packages/sdk` checks the tests against the SDK's types as well as the sources. [`scripts/mutation`](scripts/mutation/README.md) holds the harnesses that measure whether these tests would catch a deliberate break in the contracts or deploy scripts.

Merges to `main` carry `[skip ci]`, to save CI minutes, so the push to `main` does not run the suites again. A change to code runs them on its pull request and merges only once every job there has passed; a docs-only change is checked locally with `node scripts/check-docs.mjs`. Each is checked against the `main` its branch started from, so two changes merged close together can still break `main`. That happened once, on 11 November 2026, and the next merge fixed it. The rule that every job must pass dates from 14 November 2026: before it, #281, #282 and #288 merged with their oracle job failing, and #283 and #297 fixed the tests that failed. On 13 November the oracle and SDK suites were run on `main` at `eaccb91` and passed.

Expand Down
4 changes: 2 additions & 2 deletions docs/whitepaper.md
Original file line number Diff line number Diff line change
Expand Up @@ -633,9 +633,9 @@ roles, and administers `UPGRADER_ROLE`), and the timelocked role grants describe
apply to a fresh deployment, not to the testnet.
`node scripts/record-implementations.mjs --check` shows what is live.

**Test coverage**, as of 15 November 2026: 815 contract tests across 36 Foundry test files
**Test coverage**, as of 16 November 2026: 815 contract tests across 36 Foundry test files
including fuzz and invariant tests (the two slash drill suites fork Arc testnet themselves, and
the twelve in `SplitAuthorityFork` skip without `--fork-url`), 1546 oracle tests and 392 in the SDK (9 skip
the twelve in `SplitAuthorityFork` skip without `--fork-url`), 1549 oracle tests and 392 in the SDK (9 skip
outside CI and without a live deployment; every test that runs passes). CI runs Foundry, the oracle and
SDK suites, and Slither static analysis on every push, plus a documentation check that
fails when the site states something the contracts do not.
Expand Down
2 changes: 1 addition & 1 deletion oracle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -955,7 +955,7 @@ and then its silence means nothing. The slash keeper (`keeper.js`) has its own s

| Variable | Description |
|----------|-------------|
| `RPC_URL` | Ethereum JSON-RPC endpoint for the target chain. A request gives up after 30 s, and ethers does not retry it. Reads are retried by `readWithBackoff`, most of them four tries with waits of 0.5, 1 and 2 s. A refusal for the node's rate limit gets seven, each wait between half and all of a step that doubles to 8 s, so about 24 s at most |
| `RPC_URL` | Ethereum JSON-RPC endpoint for the target chain. A request gives up after 30 s, and ethers does not retry it. Reads are retried by `readWithBackoff`, most of them four tries with waits of 0.5, 1 and 2 s. A refusal for the node's rate limit gets seven, each wait between half and all of a step that doubles to 8 s, so about 24 s at most. An agent whose first read or guard read is still refused is read once more, 10 s later, before anything is written for it, and only then left for the next epoch |
| `ORACLE_PRIVATE_KEY` | Private key for the oracle wallet (must have ORACLE_ROLE). Required unless `ORACLE_PRIVATE_KEY_FILE` is set |
| `ORACLE_PRIVATE_KEY_FILE` | Path to a file holding that key, in place of `ORACLE_PRIVATE_KEY`; setting both stops the process. Prefer it in production: an environment variable shows in `docker inspect`, in `/proc/<pid>/environ` and in every child process, while a compose secret mounts the file read-only. A malformed key is refused with a fixed message that never contains the value |
| `IDENTITY_ADDRESS` | SigvaraIdentity contract address |
Expand Down
9 changes: 8 additions & 1 deletion oracle/epoch-policy.js
Original file line number Diff line number Diff line change
Expand Up @@ -378,8 +378,15 @@ function chunkForReads(readsPerItem) {
// refused at all. It took 42 seconds rather than 41, because retrying is slower than waiting.
const READ_PACE_MS = 1_500;

// How long the epoch waits before reading again an agent whose read the node refused through every retry, before
// anything was written for it. On 16 November 2026 the checker's node refused one chunk of ten guard reads for about
// 18 s, longer than readWithBackoff's schedule for a rate limit (about 24 s at most, mostly spent waiting), and the
// one read that ran out cost its agent the hour. Read once more after this pause, it has usually passed; refused
// again, the agent waits for the next epoch as before.
const REREAD_PAUSE_MS = 10_000;

module.exports = {
decideAction, decideCheckerAction, checkerGraceMs, finalizableAtOf, scoreDivergence, mapChunked, chunkForReads, READ_PACE_MS, scoringIsStale,
decideAction, decideCheckerAction, checkerGraceMs, finalizableAtOf, scoreDivergence, mapChunked, chunkForReads, READ_PACE_MS, REREAD_PAUSE_MS, scoringIsStale,
scoreRefreshSeconds, writeDueAt, writeOrder, DEFAULT_SCORE_REFRESH_HOURS, DEFAULT_MAX_WRITES_PER_EPOCH,
epochIntervalError, divergenceToleranceError, clockSkewError, MAX_TIMER_MS, MIN_EPOCH_MS,
SCORE_FACTORS, DEFAULT_DIVERGENCE_TOLERANCE, MAX_DIVERGENCE_TOLERANCE,
Expand Down
42 changes: 34 additions & 8 deletions oracle/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ const { computeScore } = require('./scoring');
const {
decideAction, decideCheckerAction, checkerGraceMs, finalizableAtOf, scoreDivergence,
epochIntervalError, divergenceToleranceError, DEFAULT_DIVERGENCE_TOLERANCE,
mapChunked, chunkForReads, READ_PACE_MS, clockSkewError, scoringIsStale,
mapChunked, chunkForReads, READ_PACE_MS, REREAD_PAUSE_MS, clockSkewError, scoringIsStale,
scoreRefreshSeconds, writeDueAt, writeOrder, DEFAULT_MAX_WRITES_PER_EPOCH,
} = require('./epoch-policy');
const { createBondWatch } = require('./bond-watch');
Expand Down Expand Up @@ -1761,7 +1761,7 @@ async function runEpochInner() {
if (bondCheck.error) console.warn(`[oracle] could not read what changed agents' bonds, so every agent is read this epoch: ${bondCheck.error.message}`);
const notRead = agents.filter((a) => bondWatch.excluded(a.didHash));
const PHASE_ONE_CHUNK = chunkForReads(gatingActive ? 4 : 3);
const agentInfos = await mapChunked(notRead.length ? agents.filter((a) => !bondWatch.excluded(a.didHash)) : agents, PHASE_ONE_CHUNK, async ({ didHash }) => {
const readFirst = async ({ didHash }) => {
try {
const [info, pending, reputation, covered] = await Promise.all([
chain.getAgentInfo(didHash),
Expand All @@ -1775,7 +1775,17 @@ async function runEpochInner() {
} catch (err) {
return { didHash, registeredAt: 0, status: -1, pending: null, reputation: null, covered: false, error: err };
}
}, READ_PACE_MS);
};
const agentInfos = await mapChunked(notRead.length ? agents.filter((a) => !bondWatch.excluded(a.didHash)) : agents, PHASE_ONE_CHUNK, readFirst, READ_PACE_MS);
// An agent whose reads the node refused through every retry is read once more after REREAD_PAUSE_MS, instead of
// waiting for the next epoch: a burst of refusals usually passes in seconds. Nothing has been written for it yet.
const refused = agentInfos.filter((info) => info.error && chain.isTransientReadError(info.error));
if (refused.length) {
console.warn(`[oracle] the node refused the first reads of ${refused.length} agent(s); reading them again in ${REREAD_PAUSE_MS / 1000} s`);
await new Promise((r) => setTimeout(r, REREAD_PAUSE_MS));
const again = new Map((await mapChunked(refused, PHASE_ONE_CHUNK, readFirst, READ_PACE_MS)).map((info) => [info.didHash, info]));
agentInfos.forEach((info, i) => { if (again.has(info.didHash)) agentInfos[i] = again.get(info.didHash); });
}

// Refund reads for payers' held negatives, before any agent's evidence is chosen (ADR 0011). Also for an agent not
// read for want of a bond, so that a negative is settled on the same reads whether or not its agent is bonded.
Expand Down Expand Up @@ -1822,11 +1832,20 @@ async function runEpochInner() {
const unbonded = [];
// The oldest scores on chain first, so the write budget goes to them (writeOrder).
const queue = writeOrder(candidates);

for (let c = 0; c < queue.length; c += GUARD_CHUNK) {
// The first of these follows the last chunk of phase 1, so it waits too.
await new Promise((r) => setTimeout(r, READ_PACE_MS));
const chunk = queue.slice(c, c + GUARD_CHUNK);
// An agent whose guard read the node refused through every retry goes back on the end of the queue, once, and is read
// again after the rest and at least REREAD_PAUSE_MS after the refusal. The guards come before any write for the agent,
// so reading it again repeats nothing. Refused a second time, it waits for the next epoch as before.
const firstPass = queue.length;
const putBack = new Set();
let putBackAt = 0;

for (let c = 0; c < queue.length;) {
// The first of these follows the last chunk of phase 1, so it waits too. A chunk of agents put back waits out the
// node's refusals, and no chunk holds both kinds.
const again = c >= firstPass;
await new Promise((r) => setTimeout(r, again ? Math.max(READ_PACE_MS, putBackAt + REREAD_PAUSE_MS - Date.now()) : READ_PACE_MS));
const chunk = queue.slice(c, again ? c + GUARD_CHUNK : Math.min(c + GUARD_CHUNK, firstPass));
c += chunk.length;
// One batch for the whole chunk: ethers folds calls issued in the same tick into a
// single JSON-RPC request, so this is one round trip rather than 2 x chunk.
const guards = await Promise.all(chunk.map(async a => {
Expand All @@ -1847,6 +1866,13 @@ async function runEpochInner() {
// The score on chain once this epoch's finalize, if any, has landed: what a new score is compared with (writeDueAt).
let onChain = reputation;
if (guardError) {
if (chain.isTransientReadError(guardError) && !putBack.has(didHash)) {
putBack.add(didHash);
putBackAt = Date.now();
queue.push(chunk[k]);
console.warn(`[oracle] ${didHash.slice(0, 10)}… the node refused its guard reads (${guardError.code || 'error'}); reading it again after the others`);
continue;
}
console.error(`[oracle] ${didHash.slice(0, 10)}… error: ${guardError.message}`);
continue;
}
Expand Down
49 changes: 49 additions & 0 deletions oracle/robustness.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -973,6 +973,55 @@ test('an agent slashed since the batch read loses its divergence records, as one
});
});

// A read the node refused through every retry, as on the checker on 16 November 2026: the guard read of one agent in a
// chunk of ten ran out of tries and the agent waited an hour. fast-retry.js cuts the epoch's pauses to milliseconds.
const reads = (calls, didHash) => calls.filter((c) => c.name === 'agentRead' && c.args[0] === didHash).map((c) => c.args[1]);
const proposed = (calls) => calls.filter((c) => c.name === 'proposeScore').map((c) => c.args[0]);

test('an agent whose guard read the node refuses through every retry is read again after the others, and written', async () => {
const world = new World();
const refused = world.addRegisteredAgent();
const other = world.addRegisteredAgent();
// The batch read answers; the guard read just before the write is refused.
world.data.agents[refused.didHash].refusedReads = [2];
world.flush();

await inEpoch({ world, env: BASE_ENV, preload: ['fast-retry.js'] }, ({ calls, logs }) => {
assert.match(logs, new RegExp(`${refused.didHash.slice(0, 10)}… the node refused its guard reads \\(SERVER_ERROR\\); reading it again after the others`));
assert.doesNotMatch(logs, /error: exceeded maximum retry limit/);
assert.deepEqual(reads(calls, refused.didHash), [1, 2, 3], 'read a third time');
assert.deepEqual(proposed(calls), [other.didHash, refused.didHash], 'and written after the agent beside it');
});
});

test('an agent whose guard read is refused again after the pause waits for the next epoch, put back only once', async () => {
const world = new World();
const agent = world.addRegisteredAgent();
world.data.agents[agent.didHash].refusedReads = [2, 3];
world.flush();

await inEpoch({ world, env: BASE_ENV, preload: ['fast-retry.js'] }, ({ calls, logs }) => {
assert.equal((logs.match(/the node refused its guard reads/g) || []).length, 1);
assert.match(logs, new RegExp(`${agent.didHash.slice(0, 10)}… error: exceeded maximum retry limit`));
assert.deepEqual(reads(calls, agent.didHash), [1, 2, 3], 'not read a fourth time');
assert.deepEqual(proposed(calls), []);
});
});

test('an agent whose first read the node refuses through every retry is read again after a pause, then written', async () => {
const world = new World();
const agent = world.addRegisteredAgent();
world.data.agents[agent.didHash].refusedReads = [1];
world.flush();

await inEpoch({ world, env: BASE_ENV, preload: ['fast-retry.js'] }, ({ calls, logs }) => {
assert.match(logs, /the node refused the first reads of 1 agent\(s\); reading them again in 10 s/);
assert.doesNotMatch(logs, /error: exceeded maximum retry limit/);
assert.deepEqual(reads(calls, agent.didHash), [1, 2, 3], 'the first read again, then the guard read');
assert.deepEqual(proposed(calls), [agent.didHash]);
});
});

test('a throttled finalize is counted once, as a throttled finalize, and nothing is proposed over it', async () => {
const world = new World();
const agent = world.addRegisteredAgent();
Expand Down
5 changes: 3 additions & 2 deletions oracle/test-support/fast-retry.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
'use strict';

// Loaded with --require beside fast-interval.js, for a process test that waits through several runs, each of which
// fails only after readWithBackoff's whole schedule (about 24 s for a node refusing for its rate limit). A wait of
// 250 ms or more is cut to 10 ms; shorter ones are left alone.
// fails only after readWithBackoff's whole schedule (about 24 s for a node refusing for its rate limit), and by an
// oracle test that waits out the epoch's pauses (READ_PACE_MS, REREAD_PAUSE_MS). A wait of 250 ms or more is cut to
// 10 ms; shorter ones are left alone. Repeating timers are untouched, so the oracle's epochs keep their interval.
const realSetTimeout = global.setTimeout;
global.setTimeout = (fn, ms, ...args) => realSetTimeout(fn, ms >= 250 ? 10 : ms, ...args);
15 changes: 10 additions & 5 deletions oracle/test-support/oracle-harness.js
Original file line number Diff line number Diff line change
Expand Up @@ -191,12 +191,16 @@ function readCalls(dir) {
// "epoch done" and then looked for the error logged just before it could fail to find it. In a file the
// lines stay in the order they were written, and a read sees everything written so far. The file sits
// beside `dir`, not in it, because a test lists the files in the oracle's own directory.
function spawnOracle(dir, env) {
function spawnOracle(dir, env, preload = []) {
const logFile = `${dir}.log`;
const fd = fs.openSync(logFile, 'a');
const child = spawn(
process.execPath,
['--require', path.join(__dirname, 'stub-chain.js'), path.join(ORACLE_DIR, 'index.js')],
[
'--require', path.join(__dirname, 'stub-chain.js'),
...preload.flatMap((name) => ['--require', path.join(__dirname, name)]),
path.join(ORACLE_DIR, 'index.js'),
],
{ cwd: dir, env, stdio: ['pipe', fd, fd] },
);
fs.closeSync(fd);
Expand All @@ -220,15 +224,16 @@ function spawnOracle(dir, env) {
* @param {World} opts.world
* @param {object} [opts.env] overrides for the child's environment
* @param {object} [opts.files] files to put in its directory first, by name
* @param {string[]} [opts.preload] more files from test-support/ to load with --require, such as fast-retry.js
*/
async function startOracle({ world, env = {}, files }, binds = 3) {
async function startOracle({ world, env = {}, files, preload = [] }, binds = 3) {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sigvara-e2e-'));
const worldFile = path.join(dir, 'world.json');
world.file = worldFile;
world.flush();
seedFiles(dir, files);
const port = await freePort();
const { child, out, exited } = spawnOracle(dir, { ...baseEnv(dir, port, worldFile), ...env });
const { child, out, exited } = spawnOracle(dir, { ...baseEnv(dir, port, worldFile), ...env }, preload);

const deadline = Date.now() + 20_000;
// Its own word first. freePort has to let go of the port to hand it over, and test files run
Expand All @@ -251,7 +256,7 @@ async function startOracle({ world, env = {}, files }, binds = 3) {
await exited;
out.drop();
fs.rmSync(dir, { recursive: true, force: true });
if (out.log.includes('EADDRINUSE') && binds > 1) return startOracle({ world, env, files }, binds - 1);
if (out.log.includes('EADDRINUSE') && binds > 1) return startOracle({ world, env, files, preload }, binds - 1);
throw new Error(`the oracle did not start (exit ${out.code}):\n${out.log}`);
}

Expand Down
13 changes: 13 additions & 0 deletions oracle/test-support/stub-chain.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
// agents: { "<didhash>": { operator, agentAddress, registeredAt, status, registeredBlock } }
// registeredBlock: the block it registered in, before which a read at a block finds no
// such agent (default 1)
// refusedReads: [n, ...] the reads of the agent at no block (the epoch's first read is 1, its
// guard read 2) that the node refuses for its rate limit, as ethers reports a 429 once its
// attempts run out (default none)
// standing: { "<address>": hard score of a counterparty }
// standingFails: { "<address>": true } counterparties the node cannot answer for: the read is throttled
// explodeHealth: when true, the /health route throws
Expand Down Expand Up @@ -193,6 +196,8 @@ const provider = {
const finalizedByOther = new Set();
// How many times each agent has been read at no block (agents[didHash].slashedFromRead).
const infoReads = new Map();
// The same, counted for agents[didHash].refusedReads.
const refusedCount = new Map();
// A write the world throttles (world.finalize "throttled"). Only these read as throttled, by the real
// predicate; a read that fails reaches the oracle as any other error, as it always has here.
const THROTTLED = Symbol('throttled write');
Expand All @@ -214,6 +219,14 @@ const chain = {
return { operator: ZERO, agentAddress: ZERO, registeredAt: 0, status: 0 };
}
const changed = agent.operators && asOf(agent.operators, blockTag);
if (blockTag === undefined && agent.refusedReads) {
const n = (refusedCount.get(didHash) ?? 0) + 1;
refusedCount.set(didHash, n);
record('agentRead', didHash, n);
if (agent.refusedReads.includes(n)) {
throw Object.assign(new Error('exceeded maximum retry limit'), { code: 'SERVER_ERROR', shortMessage: 'exceeded maximum retry limit' });
}
}
if (blockTag === undefined && agent.slashedFromRead) {
const n = (infoReads.get(didHash) ?? 0) + 1;
infoReads.set(didHash, n);
Expand Down
Loading