From 1ee2296d3d8e2410d760cf5c9405d17aa844099e Mon Sep 17 00:00:00 2001 From: RunTimeAdmin <248247024+RunTimeAdmin@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:47:29 -0600 Subject: [PATCH] Date the runbook table and the checklist 16 November, and say how reads retry [skip ci] The runbook's table had the operators and the watcher on 9a55028. The primary and the checker run 2ed3453 since 01:34 and 01:33 UTC on 16 November, and the watcher and the policy signer 489372c since 23:49 UTC on 15 November, read from each container. The checklist drops the line saying the publisher had not yet posted through the policy signer's hash rule. It posts every day, most recently four posts at 00:04 UTC on 16 November, each signed by the signer. The staking item gains #326's bound on slash evidence and the new Reader that follows it. The oracle README said readWithBackoff allows four attempts. Since #329 a rate-limit refusal gets seven, and the RPC_URL row now says how a read gives up and retries. The second-operator runbook's line about the old scanner is put in the past tense. Co-Authored-By: Claude Opus 5.5 --- docs/launch-checklist.md | 10 ++++------ docs/operations-runbook.md | 10 +++++----- oracle/README.md | 4 ++-- oracle/RUNBOOK-second-operator.md | 2 +- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/launch-checklist.md b/docs/launch-checklist.md index 7cba257..cb0aeb3 100644 --- a/docs/launch-checklist.md +++ b/docs/launch-checklist.md @@ -4,21 +4,19 @@ Sequenced by gate, not by date. The wider plan, adoption included, is the [roadm ## Where this stands -As of 15 November 2026. The SVR token is on Arc mainnet; the registries and the oracle run on Arc testnet only. What runs where is in the +As of 16 November 2026. The SVR token is on Arc mainnet; the registries and the oracle run on Arc testnet only. What runs where is in the [runbook](operations-runbook.md). There has been no external audit, and nobody outside the team gates on a score yet. **In progress, with dates.** - The adversarial slash drill: day 7 done on 10 November, day 21 from 24 November 03:45 UTC ([slash-drill.md](slash-drill.md), [slash-drill-log.md](slash-drill-log.md)). - The slash keepers went live on 5 November. Both hold C, and neither has yet executed a real slash. -- The publisher has not yet posted through the policy signer's new rule, which requires the feedback hash - to match its file. Three earlier real posts all pass it. **Open before mainnet.** - An external audit, and fixing what it finds (Phase 2). -- The staking half of the contract upgrade batch, on testnet after the drill: a cap on the challenge period - below the dispute period, refusing the admin as a slash's victim and a zero-amount withdrawal. The - reputation and oracle bond halves went in on 5 November. +- The staking upgrade, on testnet after the drill: a cap on the challenge period below the dispute period, + refusing the admin as a slash's victim and a zero-amount withdrawal, and a bound on slash evidence. A new + Reader follows it. The other contracts are upgraded, and [arc.md](arc.md) names each build. - The handover: moving testnet roles to a Safe and a timelock, after the drill, since `executeSlash` needs the reporter still on the committee. - A review of retention by counsel, before public outcome retrieval and peer import on mainnet. diff --git a/docs/operations-runbook.md b/docs/operations-runbook.md index 0f5e35f..c0e55dd 100644 --- a/docs/operations-runbook.md +++ b/docs/operations-runbook.md @@ -16,16 +16,16 @@ key is [incident-operator-key.md](incident-operator-key.md); bringing up an oper | State volume | `sigvara-oracle_oracle_state` | `sigvara-checker_checker_state` | | Local port, for `/metrics` | `127.0.0.1:3030` | `127.0.0.1:3031` | -**What runs what, as of 15 November 2026.** Each public `/health` names its own commit, and that is the +**What runs what, as of 16 November 2026.** Each public `/health` names its own commit, and that is the one to trust when this table has gone stale. | Service | Host | Commit | Since (UTC) | |---|---|---|---| -| Primary oracle | `archipelago-standby` | `9a55028` | 15 November, 01:34 | -| Checker | `packrai-vps` | `9a55028` | 15 November, 01:22 | +| Primary oracle | `archipelago-standby` | `2ed3453` | 16 November, 01:34 | +| Checker | `packrai-vps` | `2ed3453` | 16 November, 01:33 | | Event index | `archipelago-standby` | `b95f0aa` | 5 November, 13:31 | -| Watcher | a third host | `9a55028` | 15 November, 01:35 | -| Policy signer | `packrai-vps` | `b95f0aa` | 5 November, 13:33 | +| Watcher | a third host | `489372c` | 15 November, 23:49 | +| Policy signer | `packrai-vps` | `489372c` | 15 November, 23:49 | | ERC-8004 publisher | `archipelago-standby` | `e39ff5b` | 5 November, 11:08 | | Slash keepers | `archipelago-standby`, `packrai-vps` | `50386ef` | 10 November, 16:39 | diff --git a/oracle/README.md b/oracle/README.md index 77b1ebf..507cf27 100644 --- a/oracle/README.md +++ b/oracle/README.md @@ -955,7 +955,7 @@ and then its silence means nothing. The slash keeper (`keeper.js`) has its own s | Variable | Description | |----------|-------------| -| `RPC_URL` | Ethereum JSON-RPC endpoint for the target chain | +| `RPC_URL` | Ethereum JSON-RPC endpoint for the target chain. A request gives up after 30 s, and ethers does not retry it. Reads are retried by `readWithBackoff`, most of them four tries with waits of 0.5, 1 and 2 s. A refusal for the node's rate limit gets seven, each wait between half and all of a step that doubles to 8 s, so about 24 s at most | | `ORACLE_PRIVATE_KEY` | Private key for the oracle wallet (must have ORACLE_ROLE). Required unless `ORACLE_PRIVATE_KEY_FILE` is set | | `ORACLE_PRIVATE_KEY_FILE` | Path to a file holding that key, in place of `ORACLE_PRIVATE_KEY`; setting both stops the process. Prefer it in production: an environment variable shows in `docker inspect`, in `/proc//environ` and in every child process, while a compose secret mounts the file read-only. A malformed key is refused with a fixed message that never contains the value | | `IDENTITY_ADDRESS` | SigvaraIdentity contract address | @@ -1069,7 +1069,7 @@ A probe without the recipient filter would not do: it proves the asset and chain right and then leaves the normal case, this asset moved but not to any of our agents, indistinguishable from a broken recipient filter. -A long catch-up is what the pace is for. A cold scan from `FROM_BLOCK` on a chain this fast is hundreds of consecutive calls, and without spacing them the run is a coin flip rather than a wait: `readWithBackoff` allows four attempts, one chunk exhausting them throws out of the scan, the checkpoint is left unadvanced, and the whole range is retried next epoch. At the default pace a 255-call run takes a little over two minutes if nothing is refused, and longer in practice: a read-only probe of exactly that range on 3 Oct 2026 still had 27% of chunks refused at 500ms, because the bucket drains over a run that long even though 72 consecutive calls do not. The scan survives it where that probe would not, since every call goes through `readWithBackoff`. Pacing makes a catch-up finish; it does not make it refusal-free. +A long catch-up is what the pace is for. A cold scan from `FROM_BLOCK` on a chain this fast is hundreds of consecutive calls, and without spacing them the run is a coin flip rather than a wait: `readWithBackoff` allows four attempts, or seven for a rate limit, one chunk exhausting them throws out of the scan, the checkpoint is left unadvanced, and the whole range is retried next epoch. At the default pace a 255-call run takes a little over two minutes if nothing is refused, and longer in practice: a read-only probe of exactly that range on 3 Oct 2026 still had 27% of chunks refused at 500ms, because the bucket drains over a run that long even though 72 consecutive calls do not. The scan survives it where that probe would not, since every call goes through `readWithBackoff`. Pacing makes a catch-up finish; it does not make it refusal-free. **Enable it on every operator or on none.** Two operators scanning and one not is the same delivery asymmetry this exists to remove, pointed the other way. diff --git a/oracle/RUNBOOK-second-operator.md b/oracle/RUNBOOK-second-operator.md index 7cabee7..b7b14dc 100644 --- a/oracle/RUNBOOK-second-operator.md +++ b/oracle/RUNBOOK-second-operator.md @@ -103,7 +103,7 @@ cast call 0x3c9c12F27DDCa7048840eE3fbF0CAa1C547D8171 "bondAmount()(uint256)" --r refused 19 of 30 with `-32005`, Circle's endpoint refused 20 of 30. Individual calls are cheap, about 120ms for a 10,000-block window, so the limit is on request *rate* and not on per-call cost. The scanner used to have no pacing between chunks, which made a long - catch-up fragile: `scanRange` wraps each call in `readWithBackoff` with four attempts, so + catch-up fragile: `scanRange` wrapped each call in `readWithBackoff` with four attempts, so a few hundred consecutive chunks would probably exhaust one of them, throw, and leave the checkpoint unadvanced for a full retry next epoch.