diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e61008818..4dc16108ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ All notable changes to the DKG V10 node are documented here. The format is based ## [Unreleased] +### Fixed + +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled — which is what removes the amplification, since the peers still contacted skip the plane already served. What the curator may settle differs by plane: it settles either plane by delivering verified DATA, but it settles a plane by being EMPTY only for durable data, which it owns. Shared memory is a per-agent-address layered union contributed by many members (`PUBLIC CGs keep the union path`), so a curator holding no shared-memory rows has said nothing about the members' layers; an empty shared-memory plane is still provable, but only as a whole-round verdict once every peer has answered. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. So does a NON-curator answering `_meta` with no data — the requester itself logs "peer may have empty or pruned data graph" for that response, and without the curator present nothing can tell an empty graph from a member that has not synced it yet. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. +- **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`. +- **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. + +### Changed + +- **`sync-global` scheduler diagnostics attribute queue pressure to a trigger** (#2006): the `operation` dimension in `GET /api/diagnostics/backpressure` and in the `[backpressure]` log records changes from the work class alone (`durable`, which merely duplicated `lane`) to `:` — for example `durable:catchup-foreground` versus `durable:on-connect` or `durable:reconcile`. Both halves are closed sets, so the label space stays bounded and free of Context Graph and peer identifiers; an unrecognised source clamps to `unspecified`. Dashboards that group on `operation` for the `sync-global` scheduler will see the new values. `GET /api/sync/catchup-status` gains `result.peersNotAttempted`, the count of sync-capable peers the walk deliberately skipped. + +### Removed + +- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. `retryDelaysMs` is retained on the options type as `never`, so a caller that still sets it fails to compile rather than having it silently ignored, and both it and the removed export are rejected at RUNTIME too — an ignored `retryDelaysMs: [10]` would otherwise turn an intended 10 ms schedule into a wait of up to the full budget, measured at 41 retry attempts and 180,000 ms of blocking against the old ladder's 3 attempts and 30 ms. + + **Node operators need do nothing.** The node ships as one unit — the CLI depends on the agent as `workspace:*`, so every package moves to the same version on upgrade and no node holds a stale caller. This removal is only visible to code OUTSIDE this repository that installs `@origintrail-official/dkg-agent` from npm and calls the catch-up retry policy directly, which is an internal sync-scheduler knob rather than part of the SDK surface. Anything that does hit it gets an immediate error naming the replacement, not a silent behaviour change. + +### Operator knobs + +| Variable | Default | Effect | +| --- | --- | --- | +| `DKG_CATCHUP_STOP_ON_PROOF` | on | Set to `0`/`false`/`no`/`off` to restore the pre-#2006 full fan-out: every sync-capable peer, both requested planes, no early stop. The escape hatch for the deliberate tradeoff that foreground catch-up may land the curator's snapshot rather than the union of every peer's. | +| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend waiting for local `sync-global` capacity. An explicit `0` disables retries; a blank value is treated as unset. | +| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Unchanged. Caps in-flight per-peer sync rounds and now also caps the widest escalation wave. | + ## [10.0.11] - 2026-07-30 A focused stability release on two paths a busy node exercises constantly. One external `/api/query` read could amplify into a planner-stalling query that starved every other subsystem: a caller that had already constrained `GRAPH ?g` to a handful of verified partitions was rewritten with a second `VALUES ?g` carrying the entire allow-list, expanding a 3 KB query to roughly 24 KB and occupying the store for minutes, cascading into queue-wait timeouts across promotion, gossip validation, SWM catch-up, and durable sync. External reads now run on the store scheduler's background lane, a disconnected caller's store work is cancelled instead of orphaned, and the redundant graph rewrite is elided. Separately, the `dkg integration` CLI is brought back into line with the registry's published JSON Schema, which its parser had drifted *stricter* than — so no `manual` entry was readable at all, in either the CLI or the node dashboard's integrations sidebar. The dashboard database stays at 31 — no migration. **No smart-contract changes — no deployment required** (no Solidity source, ABI, or deployment-registry changes since 10.0.10). diff --git a/docs/use-dkg/backpressure-observability.md b/docs/use-dkg/backpressure-observability.md index 6ea8aec4a6..1e4400826c 100644 --- a/docs/use-dkg/backpressure-observability.md +++ b/docs/use-dkg/backpressure-observability.md @@ -69,6 +69,43 @@ The first registered sources are: `normal`, and `background` lanes; - `sync-global`: the process-wide sync admission queue and its sync lanes. +### Attributing `sync-global` pressure to a trigger + +The `lane` of a `sync-global` entry says *what kind of work* is queued +(`durable`, `changelog`, `shared_memory`, `swm_recovery`), but every trigger +funnels into the same few lanes. Its `operation` label therefore pairs the +collapsed work class with the **admission source** — the trigger that enqueued +it — as `:`: + +| Source | Trigger | +| --- | --- | +| `catchup-foreground` | explicit Context Graph catch-up (`POST /api/context-graph/subscribe`) | +| `catchup-background` | automatic post-approval / reconcile catch-up | +| `on-connect` | sync-on-connect after a peer dial | +| `reconcile` | the periodic sync reconciler | +| `vm-recovery` | foreground repair of specific missing Knowledge Assets | +| `swm-recovery` | curator-targeted shared-memory recovery | +| `unspecified` | a caller that did not declare an origin | + +### Tuning foreground catch-up + +Two knobs govern the foreground Context Graph catch-up that most often shows up +as `catchup-foreground` pressure. Both are read once at daemon start. + +| Variable | Default | Effect | +| --- | --- | --- | +| `DKG_CATCHUP_STOP_ON_PROOF` | on | The catch-up walks peers in escalating waves and stops once the resolved curator has settled every requested plane. Set to `0`, `false`, `no`, or `off` to restore the previous behaviour: every sync-capable peer, both requested planes, no early stop. Use this if a graph ever lands short — foreground catch-up optimises for one authoritative payload, while breadth remains the background reconcile lane's job. | +| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend being **refused** by local `sync-global` admission before the job reports a retryable `deferred`. Measured from before the first attempt, so an attempt's own queue time counts against it. It does not cancel a round the scheduler has already accepted — that one is doing real work and is bounded by `SYNC_TOTAL_TIMEOUT_MS`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. | +| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Caps in-flight per-peer sync rounds, and therefore the widest escalation wave. Raising it above the `sync-global` queue depth lets a single catch-up saturate the scheduler against itself. | + +So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}` +in a `queuedOperations` summary reads as "four explicit catch-up durable +admissions are queued, the oldest for 109 seconds", and the matching +`activeOperations` entry gives the same view for admitted work. Both halves are +closed sets, so the label space stays bounded (5 × 7) and, as before, no Context +Graph id or peer id ever reaches a metric, log line, or diagnostics response — +an unrecognized source is clamped to `unspecified`. + Other schedulers can extend `ObservableScheduler` and call its protected lifecycle methods at their existing admission boundaries. They keep complete ownership of policy. diff --git a/packages/agent/src/context-graph-meta-projection.ts b/packages/agent/src/context-graph-meta-projection.ts index 412c12cef1..d7ab1a0b12 100644 --- a/packages/agent/src/context-graph-meta-projection.ts +++ b/packages/agent/src/context-graph-meta-projection.ts @@ -128,6 +128,37 @@ const CATALOG_META_PREDICATES = new Set([ DKG_ONTOLOGY.DCT_ACCESS_RIGHTS, ]); +/** + * A `ContextGraphMetaRecord` with no facts loaded yet. + * + * Shared by every reader so a field added to the record cannot be initialized + * in one loader and forgotten in another. + */ +function emptyContextGraphMetaRecord( + contextGraphId: string, + uri: string, +): ContextGraphMetaRecord { + const isSystem = (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId); + return { + id: contextGraphId, + uri, + declared: isSystem, + isSystem, + creators: [], + curators: [], + allowedPeers: [], + allowedAgents: [], + participantAgents: [], + participantIdentityIds: [], + revokedAgents: [], + delegations: [], + subGraphs: [], + hasAgentGate: false, + hasPeerGate: false, + hasLegacyParticipantGate: false, + }; +} + export class ContextGraphMetaProjection { private readonly entries = new Map(); @@ -312,6 +343,50 @@ export class ContextGraphMetaProjection { return (await this.store.listGraphs(options)).filter((graphUri) => graphUri.startsWith(prefix)); } + /** + * Facts the Context Graph declared about ITSELF in its own `_meta` graph. + * + * `get()` deliberately unions `_meta`, AGENTS, `_catalog` and ONTOLOGY under + * first-wins precedence, which is right for privacy and listing reads — an + * AGENTS-only declaration can legitimately mark a graph private. It is NOT + * right for deciding who speaks for the graph: the merged record discards + * WHICH graph supplied each fact, so a creator contributed by AGENTS or + * `_catalog` (both of which carry THIRD-PARTY assertions — other agents' + * self-declarations and peer-fetchable catalog records) is indistinguishable + * from one the Context Graph declared about itself. + * + * Only `/_meta` is read. ONTOLOGY is deliberately NOT included even + * though a PUBLIC graph writes its definition there + * (`defGraph = isCurated ? cgMetaGraph : ontologyGraph`): ONTOLOGY is + * network-replicated, so any node can assert a `DKG_CREATOR` for a subject, + * and a row being the only one currently visible LOCALLY proves nothing + * about what the network holds. Requiring local uniqueness there would + * repeat, one graph over, the same local-cardinality fallacy that makes the + * Agent Registry route non-authoritative. + * + * The consequence is deliberate, and is a real cost: a public graph whose + * identity facts live only in replicated ONTOLOGY has NO locally trustworthy + * binding, so it earns no authority and its catch-up degrades to the previous + * bounded fan-out. That is today's behaviour rather than a regression — the + * fan-out reduction is earned by graphs that declare their own binding, and + * settling a graph on an unverifiable claim is the worse trade. + * + * Catch-up authority needs this distinction (issue #2006). + */ + async getOwnMetaFacts( + contextGraphId: string, + options: QueryOptions = {}, + ): Promise { + const uri = contextGraphDataUri(contextGraphId); + const metaGraph = contextGraphMetaGraphUri(contextGraphId); + assertSafeIri(uri); + assertSafeIri(metaGraph); + + const record = emptyContextGraphMetaRecord(contextGraphId, uri); + await this.loadContextGraphFacts(metaGraph, uri, record, options); + return record; + } + private async rebuild(contextGraphId: string, options: QueryOptions): Promise { const uri = contextGraphDataUri(contextGraphId); const ontologyGraph = contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY); @@ -330,24 +405,7 @@ export class ContextGraphMetaProjection { assertSafeIri(metaGraph); assertSafeIri(catalogGraph); - const record: ContextGraphMetaRecord = { - id: contextGraphId, - uri, - declared: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), - isSystem: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), - creators: [], - curators: [], - allowedPeers: [], - allowedAgents: [], - participantAgents: [], - participantIdentityIds: [], - revokedAgents: [], - delegations: [], - subGraphs: [], - hasAgentGate: false, - hasPeerGate: false, - hasLegacyParticipantGate: false, - }; + const record = emptyContextGraphMetaRecord(contextGraphId, uri); // Authoritative (local, fully trusted) sources first, meta-first so its // scalars win via first-wins (`??=`) precedence. The floor-filtered `_catalog` diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 9a0f8c33f6..6c514103b7 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -522,6 +522,190 @@ async function applyContextGraphListPrivacy( .map(({ policyKnown: _policyKnown, ...row }) => row); } +/** Where a resolved catch-up sync peer came from; see {@link resolveCuratorSyncPeer}. */ +export interface SyncPeerResolution { + peerId?: string; + /** + * Only `'metadata'` is AUTHORITATIVE — see {@link authoritativeSyncPeerId}. + * Everything else ranks the walk and may never end it. + * + * - `'metadata'` — the Context Graph's OWN `/_meta` declares the + * curator→peer binding, and that binding is internally + * consistent. + * - `'projection'` — the binding came from the merged metadata projection, + * which unions `_meta` with AGENTS / `_catalog` / ONTOLOGY + * and discards which graph supplied each fact. Good enough + * to rank; not a statement the graph made about itself. + * - `'registry'` — a wallet-address curator resolved through the agent + * registry, which is queried STRICTLY LOCALLY, so even a + * single local match is not proof of a network-wide binding. + * - `'bootstrap-hint'` — the authenticated join-approval hint; can be stale. + * - `'none'` — no peer at all. + */ + provenance: 'metadata' | 'projection' | 'registry' | 'bootstrap-hint' | 'none'; +} + +/** + * The peer allowed to let one answer stand for a whole Context Graph — a + * metadata-resolved curator and nothing else. A single definition so the walk's + * early-stop rule cannot be restated slightly differently at another call site. + */ +export function authoritativeSyncPeerId(resolution: SyncPeerResolution): string | undefined { + return resolution.provenance === 'metadata' ? resolution.peerId : undefined; +} + +/** + * Does this DID identify a peer directly, or does it need resolving through a + * registry? Wallet-address curators (V10) are the indirect case; a bare libp2p + * peer id (legacy) is already the answer. + */ +function curatorDidNeedsRegistryResolution(curatorIdentifier: string): boolean { + return curatorIdentifier.startsWith('0x'); +} + +/** + * Resolve the curator peer for a Context Graph together with WHERE it came from. + * + * Two routes produce a peer id here and they are NOT interchangeable: + * + * - `'metadata'` — `/_meta` names a curator DID and it resolved to a peer. + * Authoritative: that peer speaks for the whole graph. + * - `'projection'` / `'registry'` — a peer was resolved, but not from a source that + * can speak for the graph. `getCgMeta()` is a MERGED projection: it unions + * `/_meta` with the AGENTS, `_catalog` and ONTOLOGY graphs under first-wins + * precedence and discards which graph supplied each fact, so a creator + * contributed by an AGENTS-only declaration is indistinguishable from one the + * graph declared about itself. The agent registry is queried strictly locally, + * so even a unique local match is not evidence of a network-wide binding. + * Both rank the walk and neither may end it. + * - `'bootstrap-hint'` — the authenticated join-approval hint recorded in + * `preferredSyncPeers`, used while `_meta` has not arrived yet (and restored + * from the durable join-approved membership row after restart). It is a fine + * ranking signal but can be stale: peer ids are cryptographic identities, so + * a curator that rotated its libp2p key leaves an ordinary member sitting on + * the id the hint still names. + * + * Provenance is returned rather than inferred by a caller because the two + * routes routinely produce the SAME id — the join approval normally comes from + * the curator — so comparing the result against the hint cannot tell + * "metadata confirmed the curator" from "metadata found nothing and the hint + * was echoed back". Only the resolver knows which branch it took. + */ + +export async function resolveCuratorSyncPeer( + agent: DKGAgent, + /** + * The agent's `preferredSyncPeers`, passed explicitly because it is both read + * and evicted here — and because that makes the resolver directly drivable in + * a test without standing up an agent. + */ + bootstrapHints: Map, + contextGraphId: string, + options: { signal?: AbortSignal } = {}, +): Promise { + const approvedCuratorPeerId = bootstrapHints.get(contextGraphId); + const fromHint = (): SyncPeerResolution => (approvedCuratorPeerId + ? { peerId: approvedCuratorPeerId, provenance: 'bootstrap-hint' } + : { provenance: 'none' }); + + const meta = await agent.getCgMeta(contextGraphId, { signal: options.signal }); + const curatorDid = meta.curator ?? meta.curators[0] ?? ''; + // Once `_meta` identifies a curator, that authoritative route must win over + // the bootstrap hint. + if (!curatorDid) return fromHint(); + const didPrefix = 'did:dkg:agent:'; + if (!curatorDid.startsWith(didPrefix)) return fromHint(); + const curatorIdentifier = curatorDid.slice(didPrefix.length); + + // Resolve curator identifier to a peer ID. The DID value is either a + // libp2p peer ID (legacy) or an Ethereum wallet address (V10). For + // wallet addresses, prefer the deterministic DKG_CREATOR triple (which + // stores the libp2p peer ID) over the agent registry (which may return + // an arbitrary match when multiple agents register the same wallet). + let curatorPeerId = curatorIdentifier; + // Assume the weaker classification and EARN `'metadata'` below. The previous + // comment here claimed the projected `DKG_CREATOR` route came "straight out of + // `/_meta`"; it does not — `getCgMeta()` merges four graphs. + let provenance: SyncPeerResolution['provenance'] = 'projection'; + if (curatorDidNeedsRegistryResolution(curatorIdentifier)) { + let resolved = false; + + // Preferred: use the same projected metadata resolution as privacy and + // listing reads. AGENTS-only declarations can mark a graph private, so + // the refresh path must be able to discover their creator route too. + const creatorCandidates = [ + meta.creator, + ...meta.creators, + ].filter((value): value is string => Boolean(value)); + for (const creatorDid of creatorCandidates) { + if (creatorDid.startsWith(didPrefix)) { + const creatorId = creatorDid.slice(didPrefix.length); + if (!creatorId.startsWith('0x')) { + curatorPeerId = creatorId; + resolved = true; + break; + } + } + } + + // Fallback: agent registry lookup (non-deterministic if multiple agents + // share the same wallet address, but better than failing outright) + if (!resolved) { + try { + throwIfSyncAuthAborted(options.signal); + const agents = await agent.discovery.findAgents(); + throwIfSyncAuthAborted(options.signal); + const matches = agents.filter( + (a) => a.agentAddress?.toLowerCase() === curatorIdentifier.toLowerCase(), + ); + const match = matches[0]; + if (match) { + curatorPeerId = match.peerId; + resolved = true; + // NEVER authoritative, however many matches came back. `findAgents()` + // queries the LOCAL Agent Registry only, so "one match" means one match + // on this node — not that the wallet has a single registration on the + // network. Local cardinality cannot prove a binding. + provenance = 'registry'; + } + } catch { + throwIfSyncAuthAborted(options.signal); + /* registry unavailable */ + } + } + + if (!resolved) return fromHint(); + } + + // No route here earns `'metadata'`, so nothing this resolver returns may end a + // catch-up walk. That is deliberate, and it is a scope decision rather than an + // oversight — see #2006 and the follow-up issue. + // + // Earlier revisions re-derived the binding from the Context Graph's OWN + // `/_meta` graph, on the theory that reading one graph instead of the + // merged projection made the fact attributable to the graph itself. It does + // not. Source-qualifying by GRAPH establishes which graph holds the rows, not + // which WRITER supplied them: ordinary durable-meta catch-up admits + // IRI-subject descriptive metadata for the Context Graph's entity subject + // (`selectAdmittedMetadataIndexes` falls through for any predicate outside the + // control set) and inserts it verbatim into that exact `_meta` graph. A + // contacted peer can therefore supply the very rows the check reads — + // `rdf:type`, `accessPolicy`, `curator` — and manufacture its own authority. + // Tightening the SHAPE of the record (completeness, uniqueness) does not help: + // it only raises the number of rows the peer must send. + // + // Authority needs a binding from a source a peer cannot write: a + // curator-signed snapshot, an on-chain curator→peer edge, or the locally + // persisted join-approval record. None is available today — join approvals + // carry no curator signature, no chain record maps a wallet to a libp2p peer, + // and both metadata "proofs" are structural checks with no signature. Until + // one exists, every resolution ranks the walk and none ends it. + void curatorDid; + + bootstrapHints.delete(contextGraphId); + return { peerId: curatorPeerId, provenance }; +} + export class ContextGraphResolveMethods extends DKGAgentBase { async getCgMeta( this: DKGAgent, @@ -531,6 +715,22 @@ export class ContextGraphResolveMethods extends DKGAgentBase { return this.contextGraphMetaProjection.get(contextGraphId, { signal: options.signal }); } + /** + * Facts from the Context Graph's OWN `/_meta` graph only — the + * source-qualified counterpart of {@link getCgMeta}, which merges four + * graphs and discards which one supplied each fact. Used where a fact has to be attributable to + * the graph itself; see `resolveCuratorSyncPeer`. + */ + async getOwnCgMetaFacts( + this: DKGAgent, + contextGraphId: string, + options: { signal?: AbortSignal } = {}, + ): Promise { + return this.contextGraphMetaProjection.getOwnMetaFacts(contextGraphId, { + signal: options.signal, + }); + } + async listContextGraphsFromProjection(this: DKGAgent, opts?: { callerAgentAddress?: string | null }): Promise { // Before enabling this default-on: thread the caller signal into getCgMeta // and wrap per-row reads in withBudget (per A1's LIST_CONTEXT_GRAPHS_*_BUDGET_MS); @@ -1807,76 +2007,12 @@ export class ContextGraphResolveMethods extends DKGAgentBase { contextGraphId: string, options: { signal?: AbortSignal } = {}, ): Promise { - const approvedCuratorPeerId = this.preferredSyncPeers.get(contextGraphId); - const meta = await this.getCgMeta(contextGraphId, { signal: options.signal }); - const curatorDid = meta.curator ?? meta.curators[0] ?? ''; - if (!curatorDid) { - // Join approval authenticates the notification sender before recording - // this hint. It is the only curator route available during the bootstrap - // window where `_meta` has not arrived yet (and is restored from the - // durable join-approved membership row after restart). Once `_meta` - // identifies a curator, however, that authoritative route must win over - // the bootstrap hint. - return approvedCuratorPeerId; - } - const didPrefix = 'did:dkg:agent:'; - if (!curatorDid.startsWith(didPrefix)) { - return approvedCuratorPeerId; - } - const curatorIdentifier = curatorDid.slice(didPrefix.length); - - // Resolve curator identifier to a peer ID. The DID value is either a - // libp2p peer ID (legacy) or an Ethereum wallet address (V10). For - // wallet addresses, prefer the deterministic DKG_CREATOR triple (which - // stores the libp2p peer ID) over the agent registry (which may return - // an arbitrary match when multiple agents register the same wallet). - let curatorPeerId = curatorIdentifier; - if (curatorIdentifier.startsWith('0x')) { - let resolved = false; - - // Preferred: use the same projected metadata resolution as privacy and - // listing reads. AGENTS-only declarations can mark a graph private, so - // the refresh path must be able to discover their creator route too. - const creatorCandidates = [ - meta.creator, - ...meta.creators, - ].filter((value): value is string => Boolean(value)); - for (const creatorDid of creatorCandidates) { - if (creatorDid.startsWith(didPrefix)) { - const creatorId = creatorDid.slice(didPrefix.length); - if (!creatorId.startsWith('0x')) { - curatorPeerId = creatorId; - resolved = true; - break; - } - } - } - - // Fallback: agent registry lookup (non-deterministic if multiple agents - // share the same wallet address, but better than failing outright) - if (!resolved) { - try { - throwIfSyncAuthAborted(options.signal); - const agents = await this.discovery.findAgents(); - throwIfSyncAuthAborted(options.signal); - const match = agents.find( - (a) => a.agentAddress?.toLowerCase() === curatorIdentifier.toLowerCase(), - ); - if (match) { - curatorPeerId = match.peerId; - resolved = true; - } - } catch { - throwIfSyncAuthAborted(options.signal); - /* registry unavailable */ - } - } - - if (!resolved) return approvedCuratorPeerId; - } - - this.preferredSyncPeers.delete(contextGraphId); - return curatorPeerId; + return (await resolveCuratorSyncPeer( + this, + this.preferredSyncPeers, + contextGraphId, + options, + )).peerId; } async refreshMetaFromCurator( diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index c6ae75b068..51f29f893c 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -313,8 +313,10 @@ import { import { contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, orderContextGraphIdsByPriority, syncPriorityClass, + type SyncAdmissionSource, type SyncSchedulerLane, } from './sync/policy.js'; import { @@ -472,6 +474,11 @@ import { type SyncReconcilerProbe, type SyncReconcilerBackoff, } from './dkg-agent-types.js'; +import { + authoritativeSyncPeerId, + resolveCuratorSyncPeer, + type SyncPeerResolution, +} from './dkg-agent-cg-resolve.js'; import { normalizePublishContextGraphId, isPublishAsyncQuadEnvelope, @@ -939,6 +946,19 @@ export type DurableSyncOptions = { exactAssetUals?: string[]; /** Admission override for foreground VM recovery. */ priority?: number; + /** + * Which trigger asked for this sync. Recorded as a bounded dimension on + * node-wide scheduler diagnostics so queue pressure can be attributed to an + * origin. + * + * The closed union, so an ordinary in-process caller cannot introduce an + * unbounded or identifier-bearing label. The catch-up Worker RPC is the one + * path where the compile-time union guarantees nothing — a `postMessage` + * payload is whatever crossed the wire — and that edge clamps with + * `normalizeSyncAdmissionSource` in the CLI bridge before calling in. + * The scheduler re-clamps anyway, as defence in depth. + */ + source?: SyncAdmissionSource; }; type LegacyDurableContextGraphOptions = { @@ -1152,9 +1172,54 @@ export class LifecycleSyncMethods extends DKGAgentBase { lane: SyncSchedulerLane, label: string, work: () => Promise, - priorityOverride?: number, - operationSignal?: AbortSignal, + admission: { + /** Admission override for foreground catch-up / VM recovery. */ + priorityOverride?: number; + operationSignal?: AbortSignal; + /** + * Which trigger enqueued this admission. Typed as the closed union for + * ordinary callers; still normalized HERE, because this is the single + * choke point every admission passes through and a clamp that cannot be + * bypassed is worth more than one that merely type-checks. + */ + source?: SyncAdmissionSource; + } = {}, + /** + * Nothing may follow `admission`. Typed `never[]` so a TypeScript caller passing + * the old 7th positional `operationSignal` fails to compile, and captured at + * runtime so a JS one fails too — see the guard below. + */ + ...legacyPositionalArgs: never[] ): Promise { + // Before #2006 this took `(…, priorityOverride?: number, operationSignal?: AbortSignal)` + // positionally. Those collapsed into one `admission` object so the new `source` + // dimension did not become a fourth positional argument. + // + // TypeScript rejects the old shape, but a JS caller compiled against it would + // pass a number here, destructure to `undefined`, and silently lose BOTH its + // priority override AND its cancellation — an operation that ignores its abort + // signal keeps running after the caller gave up. Losing cancellation quietly is + // strictly worse than failing, so the old shape fails loudly. + // + // Deliberately NOT a compatibility shim translating the old arguments: this is an + // internal admission helper with no caller outside `packages/agent`, and a + // translated second shape would have to be carried and tested forever. + // `legacyPositionalArgs` catches the shape the 6th-argument test below cannot: + // `(…, work, undefined, signal)`. There the 6th is absent-looking and defaults to + // `{}`, so only the presence of a 7th argument reveals that a caller still thinks + // it is passing a cancellation signal. + if (legacyPositionalArgs.length > 0 + || typeof admission !== 'object' || admission === null + || typeof (admission as { aborted?: unknown }).aborted === 'boolean') { + throw new TypeError( + 'runContextGraphSyncWithBackpressure takes a single `admission` object ' + + '({ priorityOverride, operationSignal, source }). The positional ' + + 'priority/signal arguments used before issue #2006 are no longer accepted, ' + + 'because ignoring them would silently drop the caller\'s cancellation.', + ); + } + const { priorityOverride, operationSignal } = admission; + const source = normalizeSyncAdmissionSource(admission.source); const priority = priorityOverride ?? contextGraphPriority(this.config.syncContextGraphPriorities, contextGraphId); const admissionBoundary = combineSyncAdmissionSignals( @@ -1171,6 +1236,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { lane, priority, priorityClass: syncPriorityClass(priority), + source, signal: admissionBoundary.signal, logInfo: (opCtx, message) => this.log.info(opCtx, message), }, @@ -3616,6 +3682,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { this: DKGAgent, remotePeer: string, probe: SyncReconcilerProbe, + source: SyncAdmissionSource = 'on-connect', ): Promise { const lastOk = this.lastSuccessfulSyncAt.get(remotePeer); const lastProgress = this.lastSyncProgressAt.get(remotePeer); @@ -3623,7 +3690,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { try { const outcome = await this.trySyncFromPeer(remotePeer, () => { syncAccountingClearedBackoff = true; - }); + }, source); if (outcome === 'deferred-backpressure') { this.log.info( createOperationContext('sync'), @@ -3671,6 +3738,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { this: DKGAgent, remotePeer: string, onSyncAccounting?: (outcome: SyncOnConnectPeerOutcome) => void, + source: SyncAdmissionSource = 'on-connect', ): Promise { if (!this.started) { return 'not-started'; @@ -3708,12 +3776,13 @@ export class LifecycleSyncMethods extends DKGAgentBase { undefined, undefined, undefined, - { stopOnBackoffWorthyFailure: true }, + { stopOnBackoffWorthyFailure: true, source }, ), refreshMetaSyncedFlags: (contextGraphIds) => this.refreshMetaSyncedFlags(contextGraphIds), discoverContextGraphsFromStore: () => this.discoverContextGraphsFromStore(), syncSharedMemoryFromPeer: async (peerId, contextGraphIds) => this.syncSharedMemoryFromPeerDetailed(peerId, contextGraphIds, { stopOnBackoffWorthyFailure: true, + source, sharedMemorySyncPlan: await getSharedMemorySyncPlan(peerId), }), syncSharedMemoryOnConnect: syncOnConnectEnabled(this.config) && (this.config.syncSharedMemoryOnConnect ?? true), @@ -3991,7 +4060,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { if (!(await this.ensurePeerAdmittedForRecovery(peerId, ctx, 'Sync reconciler'))) continue; const shortPeer = peerId.slice(-8); this.log.info(ctx, `Sync reconciler retrying ${shortPeer} (last success: ${lastOk == null ? 'never' : `${Math.round((now - lastOk) / 1000)}s ago`}${backoff ? `, prior failures: ${backoff.failures}` : ''})`); - this.attemptSyncFromPeerWithReconcilerAccounting(peerId, probe) + this.attemptSyncFromPeerWithReconcilerAccounting(peerId, probe, 'reconcile') .then(() => undefined) .catch((err: unknown) => { const message = err instanceof Error ? err.message : String(err); @@ -4361,6 +4430,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds, onAccessDenied, options?.priority, + normalizeSyncAdmissionSource(options?.source), ); changelogResult = lane.result; legacyContextGraphIds = lane.remainingLegacyCgs; @@ -4458,8 +4528,11 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, work, - options?.priority, - operationBoundary.signal, + { + priorityOverride: options?.priority, + operationSignal: operationBoundary.signal, + source: options?.source, + }, ), operationBoundary.signal, ); @@ -4539,6 +4612,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { exactAssetUals: assetUals, stopOnBackoffWorthyFailure: true, priority: 1_000, + source: 'vm-recovery', }, ); } @@ -4736,6 +4810,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds: string[], onAccessDenied?: (contextGraphId: string) => void, priority?: number, + source?: SyncAdmissionSource, ): Promise<{ result?: DurableSyncResult; remainingLegacyCgs: string[] }> { const peerProtocols = await this.getPeerProtocols(remotePeerId); if (!peerProtocols.includes(PROTOCOL_SYNC_CHANGELOG)) { @@ -4776,7 +4851,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, run, - priority, + { priorityOverride: priority, source }, ), merge: mergeDurableSyncAccumulatorInto, markDeferred: (summary) => { @@ -5212,6 +5287,12 @@ export class LifecycleSyncMethods extends DKGAgentBase { sharedMemorySyncPlan?: SharedMemorySyncContextGraphPlan; /** Admission override for foreground catch-up. */ priority?: number; + /** + * Bounded admission origin for node-wide scheduler diagnostics. The + * closed union: the catch-up Worker RPC is the only untrusted producer + * and it clamps in the CLI bridge, while `acquire` re-clamps regardless. + */ + source?: SyncAdmissionSource; }, ): Promise { const ctx = createOperationContext('sync'); @@ -5436,7 +5517,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, run, - options?.priority, + { priorityOverride: options?.priority, source: options?.source }, ), merge: mergeSharedMemorySyncResults, markDeferred: (summary) => ({ @@ -5520,6 +5601,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { remotePeerId, contextGraphId, ), + { source: 'swm-recovery' }, ); } @@ -5865,18 +5947,18 @@ export class LifecycleSyncMethods extends DKGAgentBase { return runCatchupPlanesWithPolicy({ mode, includeSharedMemory, - syncDurable: ({ priority }) => this.syncFromPeerDetailed( + syncDurable: ({ priority, source }) => this.syncFromPeerDetailed( remotePeerId, [contextGraphId], undefined, undefined, undefined, - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ).catch(() => createFailedPeerDurableSyncResult()), - syncSharedMemory: ({ priority }) => this.syncSharedMemoryFromPeerDetailed( + syncSharedMemory: ({ priority, source }) => this.syncSharedMemoryFromPeerDetailed( remotePeerId, [contextGraphId], - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ).catch(emptyShared), }); }, @@ -6255,13 +6337,51 @@ export class LifecycleSyncMethods extends DKGAgentBase { return orderCatchupPeers(peers, preferredPeerId, privateOnly, this.knownCorePeerIds); } + /** + * Resolve the catch-up sync peer ONCE, with both notions the walk needs. + * + * They are one resolution, not two: ranking takes the best peer available + * whatever its provenance, while letting one peer's answer stand for the + * whole graph requires a metadata-resolved curator. The authenticated + * join-approval hint ranks but never settles — it can be stale, since a + * curator that rotated its libp2p key leaves an ordinary member sitting on + * the id it names. + * + * Deriving that distinction from two calls would read `_meta` twice (and run + * the registry fallback twice for a wallet-address curator) per catch-up, and + * would hide that the resolver has a side effect — it evicts the bootstrap + * hint once metadata confirms a curator, so the second call is not the same + * call as the first. + */ + async resolveSyncPeerWithProvenance( + this: DKGAgent, + contextGraphId: string, + ): Promise { + return resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId); + } + async resolvePreferredSyncPeerId(this: DKGAgent, contextGraphId: string): Promise { - // resolveCuratorPeerId consults authoritative metadata first and only then - // falls back to the authenticated join-approval hint. Calling it before - // reading preferredSyncPeers prevents that bootstrap hint from pinning all - // later catchups to a curator that metadata has superseded. - const curatorPeerId = await this.resolveCuratorPeerId(contextGraphId); - return curatorPeerId ?? this.preferredSyncPeers.get(contextGraphId); + // Deliberately NOT routed through the sibling method: each of these is one + // resolution on its own, and going through `this` would make them + // unusable against the hand-built receivers several suites call them on. + return (await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId)).peerId; + } + + /** + * The sync peer ONLY when it is a metadata-resolved curator. + * + * Provenance comes from {@link resolveCuratorSyncPeer} itself. Deriving it + * here — by comparing the resolved id against the hint — would be wrong in + * the ordinary case, where the join approval came from the curator and both + * routes name the SAME peer. + */ + async resolveAuthoritativeSyncPeerId( + this: DKGAgent, + contextGraphId: string, + ): Promise { + return authoritativeSyncPeerId( + await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId), + ); } async ensurePeerConnected(this: DKGAgent, peerId: string): Promise { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index cae5889e80..f9724c8f74 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -127,12 +127,15 @@ export type { AcceptedRfc64CatalogAccessSnapshotV1, } from './rfc64/catalog-access-policy-v1.js'; export { + SYNC_ADMISSION_SOURCES, contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, normalizeSyncContextGraphPriorities, orderContextGraphIdsByPriority, syncPriorityClass, validateSyncResponderSnapshotLimitsConfig, + type SyncAdmissionSource, type SyncContextGraphPriorityConfig, type SyncPriorityClass, type SyncResponderSnapshotLimitsConfig, @@ -222,6 +225,8 @@ export { type ImportedArtifactByteStore, type DurableSyncDiagnostics, type DurableSyncResult, + type SharedMemorySyncDiagnostics, + type SharedMemorySyncResult, } from './dkg-agent-types.js'; export { computeImportedArtifactSelector, @@ -304,18 +309,43 @@ export { // registry-scale per-peer fan-out and must be bounded by the SAME knob, without // deep-importing the compiled `dist/` module. export { mapWithConcurrency } from './map-with-concurrency.js'; -export { CATCHUP_MAX_CONCURRENT_PEER_SYNCS } from './sync/catchup-concurrency.js'; export { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, + catchupWaveSizes, +} from './sync/catchup-concurrency.js'; +// Only what a cross-package consumer genuinely needs. The CLI daemon's Worker +// catch-up runner drives the same plane policy and must not deep-import the +// compiled `dist/`; everything else here — the backoff curve, the env parser, +// the injected clock seams — is retry-policy internals, and in-package tests +// import those from `./sync/catchup-policy.js` directly rather than pinning +// them to the published surface. +export { + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, FOREGROUND_CATCHUP_SYNC_PRIORITY, catchupPriorityForMode, + catchupSourceForMode, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, + type CatchupAdmissionSource, + type CatchupBackpressureRetryPolicy, type CatchupMode, type CatchupPlaneContext, + type CatchupPlanePolicyClock, type CatchupPlanePolicyOptions, type CatchupPlanePolicyResult, type CatchupPlaneResult, } from './sync/catchup-policy.js'; +// Which peer may let one answer stand for a WHOLE Context Graph is the load- +// bearing distinction of the foreground catch-up walk (#2006), and the walk +// lives in the CLI's worker. Publishing the model — rather than letting the +// bridge re-shape it into a bare string — is what keeps the two sides from +// drifting: adding or renaming a provenance value must break the consumer, not +// silently downgrade it to "not authoritative". +export { + authoritativeSyncPeerId, + type SyncPeerResolution, +} from './dkg-agent-cg-resolve.js'; export { classifyDurableProgress, createFailedPeerDurableSyncResult, diff --git a/packages/agent/src/sync/backpressure.ts b/packages/agent/src/sync/backpressure.ts index 99ccc4b7c0..7b4eb8d7aa 100644 --- a/packages/agent/src/sync/backpressure.ts +++ b/packages/agent/src/sync/backpressure.ts @@ -1,5 +1,10 @@ import { getMetrics, type OperationContext } from '@origintrail-official/dkg-core'; -import type { SyncPriorityClass, SyncSchedulerLane } from './policy.js'; +import { + normalizeSyncAdmissionSource, + type SyncAdmissionSource, + type SyncPriorityClass, + type SyncSchedulerLane, +} from './policy.js'; import { PriorityAdmissionQueue, type PriorityAdmission, @@ -31,6 +36,7 @@ interface GlobalQueuePayload { limit: number; label: string; contextGraphId?: string; + source: SyncAdmissionSource; } export const DEFAULT_SYNC_GLOBAL_MAX_INFLIGHT = 2; @@ -50,6 +56,19 @@ function syncOperationClass(label: string): string { } } +/** + * `:` — the operation dimension of node-wide pressure + * diagnostics. The work class alone duplicates `lane`; pairing it with the + * admission source is what lets an operator attribute a saturated `sync-global` + * queue to explicit catch-up versus sync-on-connect versus reconcile, and read + * per-trigger queue/active ages straight off the snapshot. Both halves are + * closed sets (5 × 7), so the label space stays bounded and free of Context + * Graph and peer identifiers. + */ +function syncAdmissionOperation(payload: GlobalQueuePayload): string { + return `${syncOperationClass(payload.label)}:${normalizeSyncAdmissionSource(payload.source)}`; +} + let inflight = 0; let lastLimit: number | null = null; let lastQueueLimit: number | null = null; @@ -68,8 +87,9 @@ const queue = new PriorityAdmissionQueue({ observability: { scheduler: 'sync-global', // Admission labels also carry CG/peer correlation identifiers. Collapse - // them to a fixed operation class before node-wide diagnostics/logging. - operation: (entry) => syncOperationClass(entry.payload.label), + // them to a fixed operation class, paired with the bounded admission + // source, before node-wide diagnostics/logging. + operation: (entry) => syncAdmissionOperation(entry.payload), inflightLimit: (entry) => entry.payload.limit, thresholds: { degradedQueueAgeMs: DEFAULT_SYNC_PRIORITY_AGING_MS / 2, @@ -113,6 +133,7 @@ function acquire( lane: SyncSchedulerLane; priority: number; priorityClass: SyncPriorityClass; + source: SyncAdmissionSource; signal?: AbortSignal; agingThresholdMs: number; now: () => number; @@ -129,6 +150,7 @@ function acquire( limit, label: options.label, contextGraphId: options.contextGraphId, + source: options.source, }, ownerKey: 'global', lane: options.lane, @@ -251,6 +273,14 @@ export async function withGlobalSyncBackpressure( lane?: SyncSchedulerLane; priority?: number; priorityClass?: SyncPriorityClass; + /** + * Which trigger enqueued this admission. Callers normalize at the boundary + * where the value enters (`runContextGraphSyncWithBackpressure`); the clamp + * below is defence in depth for anything that reaches the scheduler by + * another route, so a bad cast can still only widen the label space to + * `unspecified`. + */ + source?: SyncAdmissionSource; signal?: AbortSignal; /** Deterministic scheduler injection; not operator configuration. */ agingThresholdMs?: number; @@ -281,6 +311,7 @@ export async function withGlobalSyncBackpressure( lane, priority, priorityClass, + source: normalizeSyncAdmissionSource(options.source), signal: options.signal, agingThresholdMs: options.agingThresholdMs ?? DEFAULT_SYNC_PRIORITY_AGING_MS, now: options.now ?? Date.now, diff --git a/packages/agent/src/sync/catchup-concurrency.ts b/packages/agent/src/sync/catchup-concurrency.ts index 83c31c0ba6..699ca00118 100644 --- a/packages/agent/src/sync/catchup-concurrency.ts +++ b/packages/agent/src/sync/catchup-concurrency.ts @@ -3,3 +3,65 @@ export const CATCHUP_MAX_CONCURRENT_PEER_SYNCS: number = (() => { const raw = Number(process.env.DKG_CATCHUP_MAX_CONCURRENT_PEERS); return Number.isInteger(raw) && raw > 0 ? raw : 4; })(); + +/** The spellings `DKG_CATCHUP_STOP_ON_PROOF` accepts as "off"; documented verbatim. */ +const CATCHUP_STOP_ON_PROOF_OFF_VALUES = ['0', 'false', 'no', 'off'] as const; + +/** + * Parse the progressive-walk kill-switch. + * + * Exported as a pure function because the constant below resolves once at + * module load, which makes the operator contract untestable in place — and the + * contract is four documented spellings plus trimming and case-folding, any one + * of which could be dropped without a single test noticing. Default is ON: + * anything unrecognised (including unset) leaves the walk enabled, so a typo + * cannot silently restore the pre-#2006 fan-out. + */ +export function resolveCatchupStopOnProof(raw: string | undefined): boolean { + const normalized = raw?.trim().toLowerCase(); + return !CATCHUP_STOP_ON_PROOF_OFF_VALUES.some((value) => value === normalized); +} + +/** + * Operator kill-switch for the progressive catch-up walk (issue #2006). + * + * With it off, foreground catch-up reverts to contacting every sync-capable + * peer in one bounded pass — the pre-fix behaviour, kept reachable because the + * walk trades breadth for cost: a peer's `complete` flag only proves it served + * its own manifest, so stopping early can land one peer's snapshot instead of + * the union of every peer's. + */ +export const CATCHUP_STOP_ON_PROOF: boolean = + resolveCatchupStopOnProof(process.env.DKG_CATCHUP_STOP_ON_PROOF); + +/** + * Escalating wave sizes for the progressive peer walk: `startWidth`, ×2, ×2, … + * capped by `maxConcurrency` and truncated to `peerCount`. + * + * With `startWidth = 1` the first wave is a single peer, which is what makes an + * authoritative first peer cost exactly one payload; doubling afterwards keeps + * the fallback tail short, because a flat wave of `maxConcurrency` would pull + * that many concurrent full payloads before any of them could prove the plane. + * + * A single-peer first wave is only justified when there IS an authority to try + * first. With no resolvable curator the head of the ranked list has no special + * claim, so callers pass `startWidth = maxConcurrency` and the walk keeps the + * previous first-round latency while still stopping early on proof. + */ +export function catchupWaveSizes( + peerCount: number, + maxConcurrency: number, + startWidth = 1, +): number[] { + const cap = Number.isInteger(maxConcurrency) && maxConcurrency > 0 ? maxConcurrency : 1; + const sizes: number[] = []; + let remaining = Math.max(0, Math.trunc(peerCount)); + let size = Number.isInteger(startWidth) && startWidth > 0 ? Math.min(cap, startWidth) : 1; + while (remaining > 0) { + const take = Math.min(size, remaining); + sizes.push(take); + remaining -= take; + size = Math.min(cap, size * 2); + } + return sizes; +} diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 97d001dd90..d3cd44a159 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -1,7 +1,73 @@ +import type { SyncAdmissionSource } from './policy.js'; + export type CatchupMode = 'background' | 'foreground'; export const FOREGROUND_CATCHUP_SYNC_PRIORITY = 2_000; -export const CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS = [100, 250, 500] as const; + +/** First backoff step after a foreground plane is refused by local admission. */ +export const CATCHUP_BACKPRESSURE_BASE_DELAY_MS = 250; +/** Ceiling for one backoff step; the scheduler drains in seconds, not minutes. */ +export const CATCHUP_BACKPRESSURE_MAX_DELAY_MS = 5_000; +/** Fraction of a delay that jitter may add, so parallel receivers desynchronize. */ +export const CATCHUP_BACKPRESSURE_JITTER_RATIO = 0.25; + +/** + * How long one foreground plane may keep waiting for local scheduler capacity. + * + * The previous policy was a fixed `[100, 250, 500]` ladder — 850 ms in total — + * while an admitted `sync-global` round is bounded by `SYNC_TOTAL_TIMEOUT_MS` + * (120 s) per plane, and issue #2006 measured queue waits of 87–109 s. A refused + * foreground admission therefore always exhausted its budget long before the + * head of the queue could possibly have cleared. + * + * The default is deliberately set ABOVE both of those numbers: the wait has to + * outlast one full head-of-line round (120 s) plus the observed backlog, or the + * budget still gives up in exactly the saturation case it exists to survive. + * Waiting costs a timer and no work. It stays bounded, so a permanently + * saturated node fails the catch-up job with a retryable status instead of + * pinning it at `running` forever. + */ +export const DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = 180_000; + +/** + * Parse the operator-facing retry budget. + * + * Exported as a pure function because the constant below is resolved once at + * module load, which makes the env contract untestable in place — and the + * contract has a sharp edge worth pinning: a BLANK assignment is the normal + * docker-compose / `.env` / systemd shape for "not set", but `Number('')` is + * `0`, which would silently disable retries entirely and land strictly worse + * than the fixed ladder this replaced. Blank is unset; an explicit `0` still + * means "do not retry". + */ +export function resolveCatchupBackpressureMaxWaitMs(raw: string | undefined): number { + const trimmed = raw?.trim(); + if (!trimmed) return DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + const parsed = Number(trimmed); + // `Number.isInteger` alone accepts `1e308`, which is an integer by IEEE-754 and + // a budget no operator meant. Require a SAFE integer so an unusable value falls + // back to the documented default instead of becoming an unbounded wait. + return Number.isSafeInteger(parsed) && parsed >= 0 + ? parsed + : DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; +} + +export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = + resolveCatchupBackpressureMaxWaitMs(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + +/** + * Bounded admission origin recorded on node-wide scheduler diagnostics. + * + * DERIVED from the scheduler's own closed set rather than restating its literals. + * The boundedness of this label space is load-bearing — it is a metric and log + * dimension — and the scheduler owns that contract. Restating the strings meant a + * rename in `SYNC_ADMISSION_SOURCES` would leave `catchupSourceForMode` returning a + * stale literal that the scheduler then silently clamped to `unspecified`, with + * nothing connecting the two declarations. `Extract` makes that a compile error: + * a renamed member collapses this to `never` and the returns below stop building. + */ +export type CatchupAdmissionSource = + Extract; export interface CatchupPlaneResult { deferredBackpressure?: number; @@ -9,18 +75,48 @@ export interface CatchupPlaneResult { export interface CatchupPlaneContext { priority?: number; + source?: CatchupAdmissionSource; +} + +export interface CatchupBackpressureRetryPolicy { + baseDelayMs?: number; + maxDelayMs?: number; + jitterRatio?: number; + /** Total wall-clock budget for one plane's admission retries. */ + maxWaitMs?: number; +} + +/** Deterministic seams for tests; never operator configuration. */ +export interface CatchupPlanePolicyClock { + /** + * Removed with the fixed `[100, 250, 500]` ladder it configured. + * + * Declared as `never` rather than deleted outright so a caller that still + * sets it FAILS TO COMPILE instead of having it silently ignored — an + * ignored `retryDelaysMs: [10]` would turn an intended 10 ms schedule into a + * wait of up to `CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, which is a much worse way + * to learn about the change than a type error. + * + * Operators: use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`. Tests: use `retry` + * with `now` / `wait` / `random`. + * + * @deprecated + */ + retryDelaysMs?: never; + retry?: CatchupBackpressureRetryPolicy; + wait?: (delayMs: number) => Promise; + now?: () => number; + random?: () => number; } export interface CatchupPlanePolicyOptions< TDurable extends CatchupPlaneResult, TShared extends CatchupPlaneResult, -> { +> extends CatchupPlanePolicyClock { mode: CatchupMode; includeSharedMemory: boolean; syncDurable: (context: CatchupPlaneContext) => Promise; syncSharedMemory: (context: CatchupPlaneContext) => Promise; - retryDelaysMs?: readonly number[]; - wait?: (delayMs: number) => Promise; } export interface CatchupPlanePolicyResult< @@ -35,25 +131,170 @@ export function catchupPriorityForMode(mode: CatchupMode): number | undefined { return mode === 'foreground' ? FOREGROUND_CATCHUP_SYNC_PRIORITY : undefined; } -async function runCatchupPlane( +export function catchupSourceForMode(mode: CatchupMode): CatchupAdmissionSource { + return mode === 'foreground' ? 'catchup-foreground' : 'catchup-background'; +} + +/** + * Elapsed-time source for the retry deadline. + * + * `performance.now()` is monotonic; `Date.now()` is not, and the budget here is a + * duration rather than a point in time, so it must not follow a wall-clock + * correction. Falls back to `Date.now` only if `performance` is unavailable. + */ +const monotonicNow: () => number = typeof performance?.now === 'function' + ? () => performance.now() + : Date.now; + +/** A pending backoff must never keep the process alive past `agent.stop()`. */ +function defaultWait(delayMs: number): Promise { + return new Promise((resolve) => { + const timer = setTimeout(resolve, delayMs); + if (typeof timer.unref === 'function') timer.unref(); + }); +} + +/** + * Exponential backoff with additive jitter, clamped so a sleep never runs past + * the plane's retry deadline. Returns `undefined` once no useful wait remains. + */ +export function nextCatchupBackpressureDelayMs(input: { + attempt: number; + remainingMs: number; + policy?: CatchupBackpressureRetryPolicy; + random?: () => number; +}): number | undefined { + if (input.remainingMs <= 0) return undefined; + const baseDelayMs = input.policy?.baseDelayMs ?? CATCHUP_BACKPRESSURE_BASE_DELAY_MS; + const maxDelayMs = input.policy?.maxDelayMs ?? CATCHUP_BACKPRESSURE_MAX_DELAY_MS; + const jitterRatio = input.policy?.jitterRatio ?? CATCHUP_BACKPRESSURE_JITTER_RATIO; + const exponential = Math.min(maxDelayMs, baseDelayMs * 2 ** Math.max(0, input.attempt)); + const jittered = exponential * (1 + jitterRatio * (input.random?.() ?? Math.random())); + return Math.max(1, Math.min(Math.round(jittered), input.remainingMs)); +} + +/** + * Run one catch-up plane, retrying only while LOCAL admission backpressure kept + * refusing it, until a bounded wall-clock deadline. + * + * Cancellation needs no extra plumbing: an aborted admission raises an + * `AbortError`, not a `SyncBackpressureBusyError`, so it never sets + * `deferredBackpressure` and the loop's own guard exits on the next iteration. + * + * SCOPE. The budget bounds how long this plane keeps ASKING. It does not + * preempt a round the scheduler has already ACCEPTED: once admitted, the plane + * is doing (or waiting to do) real work, and the only cancellation seam reaching + * it — `operationSignal` — aborts the whole sync, so firing it at the deadline + * would kill productive catch-up under exactly the load this exists to survive. + * An accepted round is separately bounded by `SYNC_TOTAL_TIMEOUT_MS`, and the + * queue it waits in is depth-bounded, so the wait is finite either way. Refusal + * is the only signal that means "no capacity, come back later", and refusal is + * what this retries. + */ +export async function runCatchupPlaneWithPolicy( mode: CatchupMode, run: (context: CatchupPlaneContext) => Promise, - options: Pick, 'retryDelaysMs' | 'wait'>, + options: CatchupPlanePolicyClock = {}, ): Promise { - const context = { priority: catchupPriorityForMode(mode) }; - let result = await run(context); - if (mode !== 'foreground') return result; + // `retryDelaysMs` configured the fixed [100, 250, 500] ladder that #2006 replaced + // with a wall-clock budget. Retaining it as `?: never` makes a TypeScript caller + // fail to compile — but a JS caller compiled against the old shape still passes it + // and would have it IGNORED, silently turning an intended 10 ms schedule into a wait + // of up to CATCHUP_BACKPRESSURE_MAX_WAIT_MS. Checked before the mode branch so a + // background caller is not exempt. + if ((options as { retryDelaysMs?: unknown }).retryDelaysMs !== undefined) { + throw new TypeError( + 'retryDelaysMs was removed in issue #2006; catch-up retries are now bounded by ' + + 'an absolute wall-clock budget. Use `retry.maxWaitMs` (operators: ' + + 'DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS). It is rejected rather than ignored ' + + 'because ignoring it would silently extend the wait to the full budget.', + ); + } + + const context: CatchupPlaneContext = { + priority: catchupPriorityForMode(mode), + source: catchupSourceForMode(mode), + }; + if (mode !== 'foreground') return run(context); - const retryDelaysMs = options.retryDelaysMs ?? CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS; - const wait = options.wait ?? ((delayMs: number) => new Promise((resolve) => { - setTimeout(resolve, delayMs); - })); - for (const delayMs of retryDelaysMs) { + // `wait` and `now` are ONE seam, not two independent ones. + // + // Before #2006 the loop was bounded by the fixed `retryDelaysMs` ladder, so an + // injected `wait` could return immediately and the loop still ended after three + // steps. #2006 deleted the ladder and moved the terminator onto a wall-clock + // deadline read through `now` — so a caller that injects only `wait` no longer + // has a bound: `wait` resolves instantly while `now` is the real clock, and the + // loop spins as fast as the microtask queue allows for the WHOLE budget. + // Measured against the built module: 6,873,671 attempts in a 2 s budget, with + // the macrotask queue starved throughout — roughly 700 million at the shipped + // 180 s default. A frozen `now` never terminates at all. + // + // This is not only a stale-caller hazard: `{ wait }` alone is what a NEW caller + // naturally writes to keep a test fast, and it type-checks today. Rejected here + // rather than defaulted, because silently pairing it with the real clock is the + // hang, and silently pairing it with a fake one would invent a timeline the + // caller never asked for. + // + // Placed after the background early-return on purpose: background mode never + // enters the retry loop, so injecting `wait` alone there is harmless and one + // test legitimately does it to assert the loop is not entered. + if (options.wait !== undefined && options.now === undefined) { + throw new TypeError( + 'runCatchupPlaneWithPolicy: `wait` and `now` must be injected together. ' + + 'Since issue #2006 the retry loop is bounded by a wall-clock deadline read ' + + 'through `now`, so an injected `wait` without a matching `now` spins for the ' + + 'entire budget instead of stepping a schedule.', + ); + } + + // `retry.maxWaitMs` reaches the loop without passing the env parser, so a + // `NaN` or unsafe value here would make every computed delay `NaN` — which the + // default timer treats as "as soon as possible", turning a bounded backoff into + // a spin under persistent refusal. + const configuredMaxWait = options.retry?.maxWaitMs; + if (configuredMaxWait !== undefined + && !(Number.isSafeInteger(configuredMaxWait) && configuredMaxWait >= 0)) { + throw new TypeError( + `runCatchupPlaneWithPolicy: retry.maxWaitMs must be a non-negative safe integer, got ${String(configuredMaxWait)}.`, + ); + } + + // Monotonic by default. `Date.now()` moves with the wall clock, so an NTP step + // BACKWARDS during a catch-up silently extends the advertised budget — the one + // direction a bound must not move. Tests still inject `now`, and the paired + // seam is enforced above. + const now = options.now ?? monotonicNow; + const wait = options.wait ?? defaultWait; + const maxWaitMs = options.retry?.maxWaitMs ?? CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + // Absolute deadline fixed once per plane, taken BEFORE the first admission + // attempt. The attempts themselves are what consume the wall clock — a round + // that sits in the scheduler queue and then gets refused can take seconds — + // so starting the clock after the first one would make the budget "however + // long the first attempt took, PLUS maxWaitMs" instead of a per-plane bound. + const retryUntil = now() + maxWaitMs; + let result = await run(context); + for (let attempt = 0; ; attempt += 1) { if ((result.deferredBackpressure ?? 0) === 0) return result; + const delayMs = nextCatchupBackpressureDelayMs({ + attempt, + remainingMs: retryUntil - now(), + policy: options.retry, + random: options.random, + }); + if (delayMs === undefined) return result; await wait(delayMs); + // Re-check the deadline AFTER sleeping. `nextCatchupBackpressureDelayMs` + // sized the delay against the budget that remained before the sleep, but a + // timer is a lower bound: under event-loop pressure the wake-up can land + // well past `retryUntil`, and starting a fresh admission there would spend + // scheduler capacity outside the budget this plane advertised. + // + // This declines to START an attempt; an attempt already in flight is never + // interrupted. The deliberate collateral is that capacity clearing exactly + // at or just after the deadline no longer gets one extra try. + if (now() >= retryUntil) return result; result = await run(context); } - return result; } /** @@ -68,11 +309,11 @@ export async function runCatchupPlanesWithPolicy< >( options: CatchupPlanePolicyOptions, ): Promise> { - const durable = await runCatchupPlane(options.mode, options.syncDurable, options); + const durable = await runCatchupPlaneWithPolicy(options.mode, options.syncDurable, options); if (!options.includeSharedMemory || (durable.deferredBackpressure ?? 0) > 0) { return { durable, shared: null }; } - const shared = await runCatchupPlane(options.mode, options.syncSharedMemory, options); + const shared = await runCatchupPlaneWithPolicy(options.mode, options.syncSharedMemory, options); return { durable, shared }; } diff --git a/packages/agent/src/sync/policy.ts b/packages/agent/src/sync/policy.ts index 975147370d..acd748c3ed 100644 --- a/packages/agent/src/sync/policy.ts +++ b/packages/agent/src/sync/policy.ts @@ -24,6 +24,45 @@ export type SyncSchedulerLane = | 'pre_authorization' | 'responder'; +/** + * Which trigger enqueued a `sync-global` admission. + * + * The lane says WHAT kind of work is queued; every trigger funnels into the same + * few lanes, so lane alone cannot tell an operator whether a saturated queue is + * an explicit user-driven catch-up, routine sync-on-connect, or a background + * reconcile. Issue #2006 had to reconstruct that from daemon logs. + * + * The set is deliberately closed and small: these values become metric and log + * dimensions, so cardinality is a contract, not an implementation detail. + */ +export const SYNC_ADMISSION_SOURCES = [ + 'catchup-foreground', + 'catchup-background', + 'on-connect', + 'reconcile', + 'vm-recovery', + 'swm-recovery', + 'unspecified', +] as const; + +export type SyncAdmissionSource = typeof SYNC_ADMISSION_SOURCES[number]; + +const SYNC_ADMISSION_SOURCE_SET: ReadonlySet = new Set(SYNC_ADMISSION_SOURCES); + +/** + * Clamp an admission origin to the closed set before it becomes a diagnostic + * label. The union is compile-time only; a value crossing a worker/RPC boundary + * or arriving through a cast must never be able to widen the label space or + * smuggle a Context Graph / peer identifier into node-wide diagnostics. + */ +export function normalizeSyncAdmissionSource( + source: string | undefined, +): SyncAdmissionSource { + return source !== undefined && SYNC_ADMISSION_SOURCE_SET.has(source) + ? source as SyncAdmissionSource + : 'unspecified'; +} + const SNAPSHOT_LIMIT_PATHS = [ ['global', 'rows'], ['global', 'bytesEstimate'], diff --git a/packages/agent/test/agent.part-16.test.ts b/packages/agent/test/agent.part-16.test.ts index 2587ee8993..cceee3a2e2 100644 --- a/packages/agent/test/agent.part-16.test.ts +++ b/packages/agent/test/agent.part-16.test.ts @@ -165,18 +165,21 @@ describe('DKGAgent config — syncContextGraphs and queryAccess warning', () => }); expect(peerStoreReads).toBe(3); + // Background catch-up carries no admission priority override, but it + // does tag its origin so node-wide scheduler diagnostics can attribute + // queue pressure to a trigger (issue #2006). expect(syncFromPeerDetailed.calls.at(-1)).toEqual([ remotePeer.toString(), ['runtime-contextGraph'], undefined, undefined, undefined, - undefined, + { source: 'catchup-background' }, ]); expect(syncSharedMemoryFromPeerDetailed.calls.at(-1)).toEqual([ remotePeer.toString(), ['runtime-contextGraph'], - undefined, + { source: 'catchup-background' }, ]); expect(result.connectedPeers).toBe(1); expect(result.syncCapablePeers).toBe(1); diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts new file mode 100644 index 0000000000..e745ce05e0 --- /dev/null +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -0,0 +1,94 @@ +import { describe, expect, it } from 'vitest'; +import { + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, + catchupWaveSizes, + resolveCatchupStopOnProof, +} from '../src/sync/catchup-concurrency.js'; + +describe('catchupWaveSizes', () => { + it('starts with a single peer so a proving authority costs one payload', () => { + // The peer list arrives ranked authority-first, so wave 1 is the curator + // whenever one is resolvable. Issue #2006: the pre-fix fan-out pulled the + // whole graph from every sync-capable peer instead. + expect(catchupWaveSizes(14, 4)[0]).toBe(1); + expect(catchupWaveSizes(1, 4)).toEqual([1]); + }); + + it('escalates by doubling up to the concurrency cap', () => { + expect(catchupWaveSizes(14, 4)).toEqual([1, 2, 4, 4, 3]); + expect(catchupWaveSizes(20, 4)).toEqual([1, 2, 4, 4, 4, 4, 1]); + expect(catchupWaveSizes(7, 8)).toEqual([1, 2, 4]); + }); + + it('opens at the full cap when there is no authority to spend the first wave on', () => { + // A single-peer opening wave buys "one payload from the curator". With no + // resolvable curator it buys nothing and would just add a round-trip to the + // front of every round, so callers open at the cap instead. + expect(catchupWaveSizes(14, 4, 4)).toEqual([4, 4, 4, 2]); + expect(catchupWaveSizes(3, 4, 4)).toEqual([3]); + // startWidth can never exceed the concurrency cap. + expect(catchupWaveSizes(9, 2, 8)).toEqual([2, 2, 2, 2, 1]); + expect(catchupWaveSizes(5, 4, 0)).toEqual([1, 2, 2]); + }); + + it('never exceeds the cap or the peer count', () => { + for (const cap of [1, 2, 3, 4, 8]) { + for (const peerCount of [0, 1, 3, 5, 13, 40]) { + const sizes = catchupWaveSizes(peerCount, cap); + expect(sizes.reduce((sum, size) => sum + size, 0)).toBe(peerCount); + for (const size of sizes) { + expect(size).toBeGreaterThan(0); + expect(size).toBeLessThanOrEqual(cap); + } + } + } + }); + + it('degrades to serial waves for a non-positive cap instead of looping forever', () => { + expect(catchupWaveSizes(3, 0)).toEqual([1, 1, 1]); + expect(catchupWaveSizes(3, Number.NaN)).toEqual([1, 1, 1]); + expect(catchupWaveSizes(0, 4)).toEqual([]); + expect(catchupWaveSizes(-2, 4)).toEqual([]); + }); + + it('resolves the shared fan-out cap to a positive integer', () => { + // Deliberately NOT asserting an upper bound: the constant is + // env-overridable and production applies no clamp, so pinning an arbitrary + // ceiling here would fail a validly configured node + // (`DKG_CATCHUP_MAX_CONCURRENT_PEERS=32`) while proving nothing about the + // code. The real contract is the parse: a positive integer, else the + // default. + expect(Number.isInteger(CATCHUP_MAX_CONCURRENT_PEER_SYNCS)).toBe(true); + expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); + }); +}); + +describe('resolveCatchupStopOnProof', () => { + // The kill-switch is operator-facing and documented with four disabled + // spellings. `CATCHUP_STOP_ON_PROOF` resolves once at module load, so without + // a pure parser only the spelling the suite happens to set is ever exercised + // — dropping `'false'` would leave an operator who set it silently running + // the very fan-out they turned off, with every test still green. + it.each(['0', 'false', 'no', 'off'])('treats %s as off', (value) => { + expect(resolveCatchupStopOnProof(value)).toBe(false); + }); + + it.each([' off ', 'OFF', 'False', 'No\t', ' 0'])('normalizes case and surrounding space in %j', (value) => { + expect(resolveCatchupStopOnProof(value)).toBe(false); + }); + + it.each([undefined, '', ' ', '1', 'true', 'yes', 'on', 'nope', 'offf', '0.0'])( + 'leaves the walk ON for %j', + (value) => { + // Default-on is the safe direction: an unrecognised value or a typo must + // not silently restore the pre-#2006 fan-out. + expect(resolveCatchupStopOnProof(value)).toBe(true); + }, + ); + + it('resolves the module constant through the same parser', () => { + expect(CATCHUP_STOP_ON_PROOF) + .toBe(resolveCatchupStopOnProof(process.env.DKG_CATCHUP_STOP_ON_PROOF)); + }); +}); diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index 59ca4505e1..69ba515e31 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -1,22 +1,54 @@ import { describe, expect, it, vi } from 'vitest'; import { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, + CATCHUP_BACKPRESSURE_BASE_DELAY_MS, + CATCHUP_BACKPRESSURE_MAX_DELAY_MS, + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + resolveCatchupBackpressureMaxWaitMs, FOREGROUND_CATCHUP_SYNC_PRIORITY, + nextCatchupBackpressureDelayMs, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, } from '../src/sync/catchup-policy.js'; +/** + * A virtual clock whose `wait` advances the clock by exactly the requested + * delay. This makes the wall-clock retry budget deterministic: the number of + * attempts is a pure function of the injected budget and the backoff curve. + */ +function virtualClock(startMs = 1_000) { + let nowMs = startMs; + const waits: number[] = []; + return { + waits, + now: () => nowMs, + wait: async (delayMs: number) => { + waits.push(delayMs); + nowMs += delayMs; + }, + /** Charge time to something other than a backoff sleep — an attempt itself. */ + advance: (deltaMs: number) => { + nowMs += deltaMs; + }, + elapsed: () => nowMs - startMs, + }; +} + describe('runCatchupPlanesWithPolicy', () => { - it('derives foreground priority and retries durable before starting SWM', async () => { + it('derives foreground priority and source and retries durable before starting SWM', async () => { const order: string[] = []; const priorities: Array = []; - const waits: number[] = []; - const syncDurable = vi.fn(async ({ priority }: { priority?: number }) => { + const sources: Array = []; + const clock = virtualClock(); + const syncDurable = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); order.push(`durable-${syncDurable.mock.calls.length}`); return { deferredBackpressure: syncDurable.mock.calls.length === 1 ? 1 : 0 }; }); - const syncSharedMemory = vi.fn(async ({ priority }: { priority?: number }) => { + const syncSharedMemory = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); order.push('shared'); return { deferredBackpressure: 0 }; }); @@ -26,8 +58,9 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - retryDelaysMs: [3, 5], - wait: async (delayMs) => { waits.push(delayMs); }, + now: clock.now, + wait: clock.wait, + random: () => 0, }); expect(result).toEqual({ @@ -40,10 +73,16 @@ describe('runCatchupPlanesWithPolicy', () => { FOREGROUND_CATCHUP_SYNC_PRIORITY, FOREGROUND_CATCHUP_SYNC_PRIORITY, ]); - expect(waits).toEqual([3]); + expect(sources).toEqual([ + 'catchup-foreground', + 'catchup-foreground', + 'catchup-foreground', + ]); + expect(clock.waits).toEqual([CATCHUP_BACKPRESSURE_BASE_DELAY_MS]); }); it('retries only SWM when durable already completed', async () => { + const clock = virtualClock(); const syncDurable = vi.fn(async () => ({ deferredBackpressure: 0 })); const syncSharedMemory = vi.fn() .mockResolvedValueOnce({ deferredBackpressure: 1 }) @@ -54,8 +93,8 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - retryDelaysMs: [1], - wait: async () => {}, + now: clock.now, + wait: clock.wait, }); expect(result.shared?.deferredBackpressure).toBe(0); @@ -63,7 +102,12 @@ describe('runCatchupPlanesWithPolicy', () => { expect(syncSharedMemory).toHaveBeenCalledTimes(2); }); - it('returns the final durable deferral without starting dependent SWM', async () => { + it('retries a deferred plane on a wall-clock budget, not a fixed attempt count', async () => { + // Before #2006 the budget was the fixed ladder [100, 250, 500] — exactly + // four attempts totalling 850 ms — against measured sync-global queue waits + // of 87-109 s, so a refused foreground admission could never outlast the + // head of the queue. The budget is now wall-clock. + const clock = virtualClock(); const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); const syncSharedMemory = vi.fn(async () => ({ deferredBackpressure: 0 })); @@ -72,21 +116,137 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - wait: async () => {}, + retry: { maxWaitMs: 90_000 }, + now: clock.now, + wait: clock.wait, + random: () => 0, }); expect(result.durable.deferredBackpressure).toBe(1); expect(result.shared).toBeNull(); - expect(syncDurable).toHaveBeenCalledTimes( - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS.length + 1, - ); expect(syncSharedMemory).not.toHaveBeenCalled(); + // Far past the four attempts the fixed ladder allowed. + expect(syncDurable.mock.calls.length).toBeGreaterThan(4); + // …and bounded by the budget rather than running forever. + expect(clock.elapsed()).toBeLessThanOrEqual(90_000); + expect(clock.waits.reduce((sum, value) => sum + value, 0)).toBe(clock.elapsed()); + }); + + it('starts the budget before the first attempt, not after it', async () => { + // An attempt is not free: it can sit in the sync-global queue for seconds + // before being refused. Taking the deadline AFTER the first attempt made + // the real bound "however long that attempt took, PLUS maxWaitMs" — the one + // thing an operator setting a wall-clock budget is not asking for. + const clock = virtualClock(); + const attemptCostMs = 400; + const maxWaitMs = 1_000; + const syncDurable = vi.fn(async () => { + clock.advance(attemptCostMs); + return { deferredBackpressure: 1 }; + }); + const startedAt = clock.now(); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs }, + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + // At most ONE in-flight attempt may overrun the deadline — the policy + // cannot preempt a round it has already started. With the clock taken after + // the first attempt this lands at 1800 ms against a 1000 ms budget. + expect(clock.now() - startedAt).toBeLessThanOrEqual(maxWaitMs + attemptCostMs); + expect(syncDurable).toHaveBeenCalledTimes(2); + }); + + it('does not start a fresh attempt when the timer wakes past the deadline', async () => { + // A timer is a LOWER bound. `nextCatchupBackpressureDelayMs` sizes the sleep + // against the budget remaining BEFORE it, so under event-loop pressure the + // wake-up can land past `retryUntil` — and starting another admission there + // spends sync-global capacity outside the budget this plane advertised. + // + // The clock below models exactly that: every sleep overruns its request. + const overshootMs = 5_000; + let nowMs = 1_000; + const clock = { + now: () => nowMs, + wait: async (delayMs: number) => { nowMs += delayMs + overshootMs; }, + }; + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 1_000 }, + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + // The opening attempt, and nothing after the overrun. Without the post-sleep + // deadline check this is 2 — the loop sleeps once, wakes 5 s past a 1 s + // budget, and admits anyway. + expect(syncDurable).toHaveBeenCalledTimes(1); + }); + + it('never sleeps past the retry deadline', async () => { + const clock = virtualClock(); + const deadlineMs = 1_000; + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + const startedAt = clock.now(); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: deadlineMs }, + now: clock.now, + wait: clock.wait, + random: () => 1, + }); + + expect(clock.elapsed()).toBeLessThanOrEqual(deadlineMs); + expect(clock.now() - startedAt).toBeLessThanOrEqual(deadlineMs); + expect(syncDurable.mock.calls.length).toBeGreaterThan(1); }); - it('keeps background catch-up best-effort without retries or priority', async () => { + it('gives up immediately when no retry budget remains', async () => { + const clock = virtualClock(); + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + + const result = await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 0 }, + now: clock.now, + wait: clock.wait, + }); + + expect(result.deferredBackpressure).toBe(1); + expect(syncDurable).toHaveBeenCalledTimes(1); + expect(clock.waits).toEqual([]); + }); + + it('retries only planes deferred by local admission backpressure', async () => { + // A timeout or transport failure is not scheduler pressure: retrying it + // here would multiply exactly the traffic issue #2006 is about. + const clock = virtualClock(); + const syncDurable = vi.fn(async () => ({ + deferredBackpressure: 0, + timedOutPhases: 1, + failedPeers: 1, + })); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 600_000 }, + now: clock.now, + wait: clock.wait, + }); + + expect(syncDurable).toHaveBeenCalledTimes(1); + expect(clock.waits).toEqual([]); + }); + + it('keeps background catch-up best-effort without retries, priority, or a foreground source', async () => { const priorities: Array = []; - const syncDurable = vi.fn(async ({ priority }: { priority?: number }) => { + const sources: Array = []; + const syncDurable = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); return { deferredBackpressure: 1 }; }); const syncSharedMemory = vi.fn(async () => ({ deferredBackpressure: 0 })); @@ -96,6 +256,7 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, + retry: { maxWaitMs: 600_000 }, wait: async () => { throw new Error('background mode must not wait'); }, }); @@ -104,5 +265,263 @@ describe('runCatchupPlanesWithPolicy', () => { expect(syncDurable).toHaveBeenCalledTimes(1); expect(syncSharedMemory).not.toHaveBeenCalled(); expect(priorities).toEqual([undefined]); + expect(sources).toEqual(['catchup-background']); + }); +}); + +describe('the retry budget is bounded even under bad input or a moving clock', () => { + it.each([ + ['an unsafe magnitude from the environment', '1e308'], + ['a fractional value', '12.5'], + ['a negative value', '-1'], + ['a non-number', 'soon'], + ])('falls back to the documented default for %s', (_label, raw) => { + // `Number.isInteger` alone accepts 1e308 — an integer by IEEE-754 and a + // budget nobody meant. An unusable env value must become the default, never + // an unbounded wait. + expect(resolveCatchupBackpressureMaxWaitMs(raw)) + .toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); + + it.each([ + ['NaN', Number.NaN], + ['Infinity', Number.POSITIVE_INFINITY], + ['an unsafe integer', 1e308], + ['a negative budget', -1], + ['a fractional budget', 5.5], + ])('rejects %s supplied through the in-process retry seam', async (_label, maxWaitMs) => { + // This path bypasses the env parser entirely. A NaN budget makes every + // computed delay NaN, which the default timer treats as "immediately" — + // turning a bounded backoff into a spin under persistent refusal. + await expect( + runCatchupPlaneWithPolicy('foreground', async () => ({ deferredBackpressure: 1 }), { + retry: { maxWaitMs: maxWaitMs as number }, + }), + ).rejects.toThrow(/non-negative safe integer/); + }); + + it('reads the deadline from a MONOTONIC clock, not the wall clock', async () => { + // The budget is a DURATION, so it must not follow an NTP correction: a + // backwards wall-clock step would silently hand back the time it rewound, + // extending the advertised budget in the one direction a bound must not move. + // + // Asserted by what production READS rather than by simulating a rollback — + // an injected `now` is honoured verbatim, so injection would bypass the very + // protection under test. + const dateNow = vi.spyOn(Date, 'now'); + try { + await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 0 } }, + ); + expect(dateNow).not.toHaveBeenCalled(); + } finally { + dateNow.mockRestore(); + } + }); +}); + +describe('the wait/now clock seam', () => { + // Before #2006 `retryDelaysMs` bounded the loop, so an injected `wait` that + // resolved instantly still terminated after three steps. The ladder is gone and + // the terminator now lives behind `now`, so injecting `wait` ALONE removes the + // bound: measured against the built module, 6,873,671 attempts in a 2 s budget + // with the macrotask queue starved — ~700 million at the shipped 180 s default. + // `{ wait }` on its own is also what a NEW caller naturally writes to keep a + // test fast, so this is a live footgun, not only a stale-caller hazard. + it('rejects an injected wait with no matching now', async () => { + await expect( + runCatchupPlaneWithPolicy('foreground', async () => ({ deferredBackpressure: 1 }), { + wait: async () => {}, + retry: { maxWaitMs: 50 }, + }), + ).rejects.toThrow(/must be injected together/); + }); + + it('allows an injected wait in background mode, which never enters the loop', async () => { + // The complement, so the guard cannot be widened into something that breaks a + // legitimate caller: background mode returns before the retry loop, so `wait` + // alone is harmless there and one test below relies on exactly that. + await expect( + runCatchupPlaneWithPolicy('background', async () => ({ deferredBackpressure: 1 }), { + wait: async () => { throw new Error('background mode must not wait'); }, + }), + ).resolves.toEqual({ deferredBackpressure: 1 }); + }); + + it('allows the paired seam, and it still terminates on the budget', async () => { + const clock = virtualClock(); + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 300 }, now: clock.now, wait: clock.wait, random: () => 0 }, + ); + expect(result.deferredBackpressure).toBe(1); + expect(clock.elapsed()).toBeLessThanOrEqual(300); + }); + + it('leaves the un-injected production path alone', async () => { + // Neither the agent nor the CLI worker injects a clock, so the default + // `Date.now` + real `setTimeout` pairing must keep working untouched. + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 0 } }, + ); + expect(result.deferredBackpressure).toBe(1); + }); +}); + +describe('the removed retryDelaysMs ladder', () => { + // Retaining it as `?: never` makes a TypeScript caller fail to compile, which the + // enforced type test pins. But TypeScript is not the runtime: a JS caller compiled + // against the pre-#2006 shape still passes it, and ignoring it would silently turn + // an intended 10 ms schedule into a wait of up to the full budget. So it is + // REJECTED, not ignored. + it.each(['foreground', 'background'] as const)('is rejected at runtime in %s mode', async (mode) => { + await expect( + runCatchupPlaneWithPolicy(mode, async () => ({ deferredBackpressure: 1 }), { + retryDelaysMs: [10, 20], + retry: { maxWaitMs: 50 }, + } as never), + ).rejects.toThrow(/retryDelaysMs was removed/); + }); + + it('leaves the supported replacement working', async () => { + const clock = virtualClock(); + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 300 }, now: clock.now, wait: clock.wait, random: () => 0 }, + ); + expect(result.deferredBackpressure).toBe(1); + expect(clock.elapsed()).toBeLessThanOrEqual(300); + }); +}); + +describe('nextCatchupBackpressureDelayMs', () => { + it('grows exponentially and clamps at the per-step ceiling', () => { + const delays = Array.from({ length: 8 }, (_, attempt) => nextCatchupBackpressureDelayMs({ + attempt, + remainingMs: Number.MAX_SAFE_INTEGER, + random: () => 0, + })); + + expect(delays[0]).toBe(CATCHUP_BACKPRESSURE_BASE_DELAY_MS); + for (let i = 1; i < delays.length; i += 1) { + expect(delays[i]!).toBeGreaterThanOrEqual(delays[i - 1]!); + } + expect(delays.at(-1)).toBe(CATCHUP_BACKPRESSURE_MAX_DELAY_MS); + }); + + it('applies jitter so parallel receivers do not retry in lockstep', () => { + const low = nextCatchupBackpressureDelayMs({ attempt: 3, remainingMs: 1e9, random: () => 0 }); + const high = nextCatchupBackpressureDelayMs({ attempt: 3, remainingMs: 1e9, random: () => 1 }); + + expect(low).toBeLessThan(high!); + expect(low).toBeGreaterThanOrEqual(CATCHUP_BACKPRESSURE_BASE_DELAY_MS); + }); + + it('never returns a delay that overshoots the remaining budget', () => { + expect(nextCatchupBackpressureDelayMs({ attempt: 10, remainingMs: 40, random: () => 1 })).toBe(40); + expect(nextCatchupBackpressureDelayMs({ attempt: 0, remainingMs: 0 })).toBeUndefined(); + expect(nextCatchupBackpressureDelayMs({ attempt: 0, remainingMs: -5 })).toBeUndefined(); + }); +}); + +describe('CATCHUP_BACKPRESSURE_MAX_WAIT_MS', () => { + it('outlasts one head-of-line round plus the queue waits it exists to survive', () => { + // Issue #2006 measured `sync-global` queue waits of 87-109 s, and an + // admitted round is itself bounded by SYNC_TOTAL_TIMEOUT_MS (120 s). A + // budget below those numbers gives up in exactly the saturation case it was + // introduced for — which is what the old fixed 850 ms ladder did. + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBeGreaterThan(120_000); + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBeGreaterThan(109_000); + }); + + it('keeps retrying past a 90-second capacity clear under the default budget', async () => { + // Virtual clock: admission stays refused until 90 s have elapsed, i.e. a + // realistic head-of-line drain. The default policy must still be retrying + // then, and must succeed rather than return deferred. + const clock = virtualClock(0); + const syncDurable = vi.fn(async () => ( + clock.now() >= 90_000 ? { deferredBackpressure: 0 } : { deferredBackpressure: 1 } + )); + + const result = await runCatchupPlaneWithPolicy('foreground', syncDurable, { + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + expect(result.deferredBackpressure).toBe(0); + expect(clock.now()).toBeGreaterThanOrEqual(90_000); + expect(clock.now()).toBeLessThanOrEqual(CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); +}); + +describe('resolveCatchupBackpressureMaxWaitMs', () => { + it('treats a blank assignment as unset, not as zero', () => { + // `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS=` in a compose file / .env / unit + // file is the normal shape for "not set". `Number('')` is 0, so a naive + // parser turns it into "never retry" — silently worse than the fixed ladder + // this policy replaced, and with no log to notice it by. + expect(resolveCatchupBackpressureMaxWaitMs('')).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + expect(resolveCatchupBackpressureMaxWaitMs(' ')).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + expect(resolveCatchupBackpressureMaxWaitMs(undefined)).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); + + it('honours an explicit zero as "do not retry"', () => { + expect(resolveCatchupBackpressureMaxWaitMs('0')).toBe(0); + expect(resolveCatchupBackpressureMaxWaitMs(' 0 ')).toBe(0); + }); + + it('honours a positive integer budget', () => { + expect(resolveCatchupBackpressureMaxWaitMs('45000')).toBe(45_000); + expect(resolveCatchupBackpressureMaxWaitMs(' 600000 ')).toBe(600_000); + }); + + it('falls back to the default for values it cannot honour', () => { + for (const raw of ['-1', '1.5', 'abc', 'NaN', 'Infinity', '1e3ms']) { + expect(resolveCatchupBackpressureMaxWaitMs(raw)).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + } + }); + + it('resolves the module constant through the same parser', () => { + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBe( + resolveCatchupBackpressureMaxWaitMs(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS), + ); + }); +}); + +describe('foreground backoff timer', () => { + it('unrefs the pending sleep so a backoff cannot outlive agent.stop()', async () => { + // The budget is minutes now. A referenced timer would hold the event loop + // open for that long past shutdown, and every other test injects `wait`, + // so the real `defaultWait` would otherwise never be exercised. + const realSetTimeout = globalThis.setTimeout; + const unref = vi.fn(); + const spy = vi.spyOn(globalThis, 'setTimeout').mockImplementation((( + handler: TimerHandler, + timeout?: number, + ...rest: unknown[] + ) => { + const handle = (realSetTimeout as any)(handler, Math.min(timeout ?? 0, 1), ...rest); + return Object.assign(handle as object, { unref }) as never; + }) as never); + + try { + let attempts = 0; + await runCatchupPlaneWithPolicy('foreground', async () => { + attempts += 1; + return { deferredBackpressure: attempts === 1 ? 1 : 0 }; + }, { retry: { maxWaitMs: 5_000 } }); + + expect(attempts).toBe(2); + expect(unref).toHaveBeenCalled(); + } finally { + spy.mockRestore(); + } }); }); diff --git a/packages/agent/test/catchup-retry-contract.typecheck.ts b/packages/agent/test/catchup-retry-contract.typecheck.ts new file mode 100644 index 0000000000..5b0363b821 --- /dev/null +++ b/packages/agent/test/catchup-retry-contract.typecheck.ts @@ -0,0 +1,99 @@ +import { + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + DKGAgent, + runCatchupPlaneWithPolicy, + type CatchupPlanePolicyClock, + type CatchupPlanePolicyOptions, + type CatchupPlaneResult, +} from '@origintrail-official/dkg-agent'; +// @ts-expect-error CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS is REMOVED from the +// package root. It named the fixed [100, 250, 500] ladder, which no longer +// exists — re-exporting it would hand a consumer a schedule the node does not +// follow. A stale caller must fail to resolve it, not compile against a lie. +import { CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS } from '@origintrail-official/dkg-agent'; + +// `retryDelaysMs` configured the fixed `[100, 250, 500]` ladder that issue #2006 +// replaced with a wall-clock budget. It is retained as `never` rather than +// deleted so that setting it is a COMPILE error instead of a silent no-op: an +// ignored `retryDelaysMs: [10]` would turn an intended 10 ms schedule into a +// wait of up to `CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, which is a far worse way to +// discover the change than a type error. +// +// That guarantee is a property of the PUBLISHED type, so it is pinned here +// rather than in a runtime test — no runtime assertion can observe it. +// +// Object literals alone CANNOT carry it. Excess-property checking rejects +// `{ retryDelaysMs: [...] }` against an annotated target whether the member is +// declared `never` or absent entirely, so a literal-only test passes in both +// worlds and proves nothing about the difference. Deleting the member is +// precisely the stale-caller silent-ignore case the `never` exists to prevent, +// so it is pinned two ways that a deletion breaks: an indexed access on the +// member itself, and a stale options object flowing through a VARIABLE, where +// excess properties are permitted and only a declared `never` can refuse them. + +// Fails to compile (TS2339) if the member is deleted rather than kept `never`. +declare const removedLadder: CatchupPlanePolicyClock['retryDelaysMs']; +// …and `undefined` is the only value it can hold. +const ladderIsUninhabited: undefined = removedLadder; + +declare const staleCallerOptions: { + retry: { maxWaitMs: number }; + retryDelaysMs: number[]; +}; +// @ts-expect-error a stale options VARIABLE carrying the removed ladder must not +// flow in structurally — this is the case excess-property checking would let by, +// and the one that silently reverted to a full-budget wait before the `never`. +const stale: CatchupPlanePolicyClock = staleCallerOptions; + +// @ts-expect-error retryDelaysMs was removed with the fixed ladder it configured. +const clock: CatchupPlanePolicyClock = { retryDelaysMs: [10, 20] }; + +const planes: CatchupPlanePolicyOptions = { + mode: 'foreground', + includeSharedMemory: false, + syncDurable: async () => ({}), + syncSharedMemory: async () => ({}), + // @ts-expect-error the same option is equally rejected on the two-plane options. + retryDelaysMs: [10], +}; + +// The replacements must stay importable and assignable, so this file cannot +// pass merely because the whole surface decayed. +const supported: CatchupPlanePolicyClock = { retry: { maxWaitMs: 5_000 } }; +const replacementBudget: number = CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + +export declare const pinned: [ + typeof clock, + typeof planes, + typeof supported, + typeof stale, + typeof ladderIsUninhabited, + typeof replacementBudget, + typeof CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, + typeof runCatchupPlaneWithPolicy, +]; + +// The admission parameters of `runContextGraphSyncWithBackpressure` collapsed from +// positional `(priorityOverride?: number, operationSignal?: AbortSignal)` into a single +// object, so the new `source` dimension did not become a fourth positional argument. +// +// The runtime guard rejects the old shape (see durable-sync-lifecycle-binding.test.ts). +// This pins the COMPILE-TIME half, and specifically the seventh-argument case: without +// the `...legacyPositionalArgs: never[]` rest parameter a stale caller passing a +// trailing AbortSignal type-checks, and TypeScript would say nothing about a caller +// that is quietly losing its cancellation. +declare const staleAdmissionCaller: DKGAgent; +declare const staleSignal: AbortSignal; + +const staleAdmissionCall = () => staleAdmissionCaller.runContextGraphSyncWithBackpressure( + {} as never, + 'cg', + 'durable' as never, + 'label', + async () => 1, + {}, + // @ts-expect-error nothing may follow `admission`; this is the pre-#2006 positional signal. + staleSignal, +); + +export declare const pinnedAdmission: typeof staleAdmissionCall; diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index 466d7d2a40..ef11b26c26 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1231,14 +1231,22 @@ describe('refreshMetaFromCurator', () => { const bootstrapPeer = 'peer-from-join-approval'; const authoritativePeer = 'peer-from-authoritative-meta'; const preferredSyncPeers = new Map([[contextGraphId, bootstrapPeer]]); + const declaredFacts = { + // A complete canonical definition, in the shape the projection produces: + // each declared fact appears as the scalar AND in its array. + declared: true, + accessPolicy: 'private', + curator: 'did:dkg:agent:0x0000000000000000000000000000000000000abc', + curators: ['did:dkg:agent:0x0000000000000000000000000000000000000abc'], + creator: `did:dkg:agent:${authoritativePeer}`, + creators: [`did:dkg:agent:${authoritativePeer}`], + }; const agent = { preferredSyncPeers, - getCgMeta: async () => ({ - curator: 'did:dkg:agent:0x0000000000000000000000000000000000000abc', - curators: [], - creator: `did:dkg:agent:${authoritativePeer}`, - creators: [], - }), + getCgMeta: async () => declaredFacts, + // The Context Graph declares this curator→peer binding in its OWN `_meta`, + // which is what makes it authoritative rather than merely rankable (#2006). + getOwnCgMetaFacts: async () => declaredFacts, discovery: { findAgents: async () => { throw new Error('creator metadata should resolve the curator peer'); @@ -1253,10 +1261,18 @@ describe('refreshMetaFromCurator', () => { expect(resolved).toBe(authoritativePeer); expect(preferredSyncPeers.has(contextGraphId)).toBe(false); - const lifecycleResolved = await LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId.call({ + // The same resolution through the lifecycle entry points, against the real + // metadata rather than a stubbed curator: the join-approved peer ranks only + // until `_meta` names someone. The declared answer then wins the RANKING — + // but it confers no authority, because `_meta` identifies the graph that + // holds the rows, not the writer that supplied them. + const lifecycleAgent = { + ...agent, preferredSyncPeers: new Map([[contextGraphId, bootstrapPeer]]), - resolveCuratorPeerId: async () => authoritativePeer, - } as never, contextGraphId); - expect(lifecycleResolved).toBe(authoritativePeer); + }; + expect(await LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId + .call(lifecycleAgent as never, contextGraphId)).toBe(authoritativePeer); + expect(await LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId + .call(lifecycleAgent as never, contextGraphId)).toBeUndefined(); }); }); diff --git a/packages/agent/test/context-graph-meta-projection.test.ts b/packages/agent/test/context-graph-meta-projection.test.ts index 46ab39e3c9..aadd3faef4 100644 --- a/packages/agent/test/context-graph-meta-projection.test.ts +++ b/packages/agent/test/context-graph-meta-projection.test.ts @@ -534,3 +534,105 @@ describe('ContextGraphMetaProjection', () => { expect((await projection.get(id)).accessPolicy).toBe('private'); }); }); + +describe('getOwnMetaFacts', () => { + const CURATOR_DID = 'did:dkg:agent:0x00000000000000000000000000000000000000ab'; + const CREATOR_DID = 'did:dkg:agent:12D3KooWCuratorPeer'; + + /** + * A Context Graph's definition is written to ONE graph, chosen by access + * policy (`dkg-agent-context-graph.ts`): + * + * const defGraph = isCurated ? cgMetaGraph : ontologyGraph; + * + * These cover the reader against a real store rather than through a stubbed + * agent, because the whole point of the reader is WHICH graph a fact came + * from — a stub cannot get that wrong, and a stub is what let an earlier + * `_meta`-only version look correct while missing every public graph. + */ + it('does NOT read ONTOLOGY, even though a public graph defines itself there', async () => { + // A public Context Graph writes its definition to ONTOLOGY, so it is + // tempting to read it here. ONTOLOGY is network-replicated, though: this + // node can hold an injected `DKG_CREATOR` for a subject WITHOUT holding the + // real one, and then "the only creator I can see" is the attacker's. Local + // cardinality proves nothing about the network — the same reason the Agent + // Registry route is non-authoritative. + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = 'own-definition-public'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + ]); + + const own = await projection.getOwnMetaFacts(id); + expect(own.curators).toEqual([]); + expect(own.creators).toEqual([]); + // The merged projection still sees them — that is the difference the + // authority decision turns on, and the reason this reader exists. + expect((await projection.get(id)).creators).toEqual([CREATOR_DID]); + }); + + it('reads a CURATED graph definition, which lives in the graph\'s own _meta', async () => { + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = '0x00000000000000000000000000000000000000ab/own-definition-curated'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphMetaGraphUri(id) }, + ]); + + const own = await projection.getOwnMetaFacts(id); + expect(own.curators).toEqual([CURATOR_DID]); + expect(own.creators).toEqual([CREATOR_DID]); + }); + + it('ignores creators contributed by AGENTS or the peer-fetchable _catalog', async () => { + // The reason this reader exists: `get()` merges these in and discards which + // graph supplied each fact, so a third-party assertion becomes + // indistinguishable from the graph's own declaration. + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = 'own-definition-third-party'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWAgentsClaim', graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.AGENTS) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWCatalogClaim', graph: contextGraphCatalogUri(id) }, + ]); + + expect((await projection.getOwnMetaFacts(id)).creators).toEqual([]); + // …while the merged projection does surface the AGENTS claim, which is the + // precise difference the authority decision turns on. + expect((await projection.get(id)).creators).toContain('did:dkg:agent:12D3KooWAgentsClaim'); + }); + + it('ignores an injected ONTOLOGY creator even when it is the ONLY one visible', async () => { + // The attack this reader exists to stop: the graph's own `_meta` has not + // synced (or names only the curator), and the sole `DKG_CREATOR` this node + // can see for the subject was asserted by someone else. A reader that took + // ONTOLOGY would hand that peer the authority to settle the whole graph on + // an empty answer. + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = '0x00000000000000000000000000000000000000ab/own-definition-injected'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWInjectedPeer', graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + ]); + + const own = await projection.getOwnMetaFacts(id); + expect(own.curators).toEqual([CURATOR_DID]); + expect(own.creators).toEqual([]); + }); +}); diff --git a/packages/agent/test/durable-sync-lifecycle-binding.test.ts b/packages/agent/test/durable-sync-lifecycle-binding.test.ts index f07c53cfce..2a4eec0c6a 100644 --- a/packages/agent/test/durable-sync-lifecycle-binding.test.ts +++ b/packages/agent/test/durable-sync-lifecycle-binding.test.ts @@ -18,6 +18,8 @@ vi.mock('../src/sync/requester/graph-scoped-materialization.js', async (importOr }; }); +import { PROTOCOL_SYNC_CHANGELOG } from '@origintrail-official/dkg-core'; +import { createDurableSyncAccumulator } from '../src/sync/durable-progress.js'; import { DKGAgent } from '../src/dkg-agent.js'; import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; import { @@ -335,10 +337,120 @@ describe('durable sync lifecycle chain binding', () => { exactAssetUals: [exactUal], stopOnBackoffWorthyFailure: true, priority: 1_000, + // The admission SOURCE is what makes this show up as `durable:vm-recovery` + // rather than `durable:unspecified` on the sync-global scheduler, which is + // the whole point of the label. Without this line, deleting it from the + // call site keeps every test green and only the Grafana attribution rots. + source: 'vm-recovery', }); expect(runLegacyDurableSync.mock.calls[0]?.[6]).not.toHaveProperty('totalTimeoutMs'); }); + it.each([ + ['a positional priority override', [2000]], + ['a positional AbortSignal', ['SIGNAL']], + // The shape the 6th-argument check alone cannot see: the 6th is absent-looking + // and defaults to `{}`, so only the PRESENCE of a 7th reveals that the caller + // still believes it is passing a cancellation signal. Before the rest-parameter + // guard this returned normally and dropped the signal. + ['a cancellation-only legacy call', [undefined, 'SIGNAL']], + ['both legacy positionals', [2000, 'SIGNAL']], + ])('rejects the pre-#2006 positional admission shape: %s', async (_label, tail) => { + // The old signature was (ctx, cg, lane, label, work, priorityOverride?, signal?). + // A JS caller compiled against it would destructure to undefined and silently + // lose its priority AND its cancellation — an operation that ignores its abort + // signal keeps running after the caller gave up. That must fail loudly. + const legacyArgs = (tail as unknown[]).map( + (a) => (a === 'SIGNAL' ? new AbortController().signal : a), + ); + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + node: { stopSignal: undefined }, + syncScheduler: { acquire: async () => ({ release: () => {} }) }, + }; + + await expect( + (LifecycleSyncMethods.prototype.runContextGraphSyncWithBackpressure as any).call( + agentLike, + {}, + 'cg-legacy', + 'durable', + 'label', + async () => 'done', + ...legacyArgs, + ), + ).rejects.toThrow(/takes a single .admission. object/); + }); + + it('labels changelog-lane admissions at the call site', async () => { + // The changelog delta lane (OT-RFC-59) is a SEPARATE production admission path + // from the durable and shared-memory ones already covered. A public Context + // Graph on a changelog-capable peer never reaches `runLegacyDurableSync`, so a + // dropped source here would surface only as `changelog:unspecified` on + // /api/diagnostics/backpressure while every existing source test stayed green. + const admissions: unknown[][] = []; + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + getPeerProtocols: async () => [PROTOCOL_SYNC_CHANGELOG], + isPrivateContextGraph: async () => false, + // Record the admission and return a real accumulator: the lane folds the + // result, so an empty object would fail inside the merge before the + // assertion below could run. + runContextGraphSyncWithBackpressure: async (...args: unknown[]) => { + admissions.push(args); + return createDurableSyncAccumulator(); + }, + }; + + await LifecycleSyncMethods.prototype.runChangelogLane.call( + agentLike as never, + ctx, + '12D3KooWChangelogPeer', + ['public-cg'], + undefined, + 2_000, + 'catchup-foreground', + ); + + expect(admissions).toHaveLength(1); + const [, contextGraphId, lane, , , admission] = admissions[0] as unknown[]; + expect(contextGraphId).toBe('public-cg'); + expect(lane).toBe('changelog'); + expect(admission).toMatchObject({ + priorityOverride: 2_000, + source: 'catchup-foreground', + }); + }); + + it('labels standalone SWM recovery admissions at the call site', async () => { + // The sibling of the VM-recovery assertion above, and the one that had NO + // coverage: a regression dropping this source would report SWM recovery + // pressure as `shared-memory:unspecified` on the sync-global scheduler. + // Asserted on the real prototype so it pins the production call site, not a + // re-statement of it. + const runContextGraphSyncWithBackpressure = vi.fn(async () => ({})); + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + runContextGraphSyncWithBackpressure, + }; + + await LifecycleSyncMethods.prototype.recoverContextGraphSwmFromPeer.call( + agentLike as any, + '12D3KooWSwmRecoveryPeer', + 'private-cg', + ); + + expect(runContextGraphSyncWithBackpressure).toHaveBeenCalledTimes(1); + const [, contextGraphId, lane, , , admission] = + runContextGraphSyncWithBackpressure.mock.calls[0] as unknown as unknown[]; + expect(contextGraphId).toBe('private-cg'); + expect(lane).toBe('swm_recovery'); + expect(admission).toEqual({ source: 'swm-recovery' }); + }); + it('honors an explicit exact-asset timeout while internal VM recovery keeps 600 seconds', async () => { vi.spyOn(Date, 'now').mockReturnValue(1_800_000_000_000); const exactUal = 'did:dkg:base:84532/0x1111111111111111111111111111111111111111/1'; diff --git a/packages/agent/test/sync-backpressure.test.ts b/packages/agent/test/sync-backpressure.test.ts index 9c43aeaaa1..382fe6b2a0 100644 --- a/packages/agent/test/sync-backpressure.test.ts +++ b/packages/agent/test/sync-backpressure.test.ts @@ -12,6 +12,7 @@ import { withGlobalSyncBackpressure, } from '../src/sync/backpressure.js'; import { PriorityAdmissionQueue } from '../src/sync/priority-admission-queue.js'; +import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; const tick = () => new Promise((resolve) => setTimeout(resolve, 0)); @@ -216,6 +217,53 @@ describe('sync global backpressure', () => { ]); }); + it('carries the admission source from the production helper through to the scheduler', async () => { + // The two halves of this contract were covered separately: the call sites + // were proven to SUPPLY a source, and `withGlobalSyncBackpressure` was proven + // to RENDER it as `:`. Nothing covered the line between + // them — the `source,` handed to `withGlobalSyncBackpressure` inside + // `runContextGraphSyncWithBackpressure`. Dropping it leaves both groups green + // while every real admission reports `durable:unspecified` on + // /api/diagnostics/backpressure, which is the attribution issue #2006 had to + // reconstruct from daemon logs. + const agentLike = { + config: { syncGlobalMaxInflight: 1, syncGlobalQueueLimit: 1 }, + node: { stopSignal: undefined }, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + }; + + let releaseWork!: () => void; + const admitted = LifecycleSyncMethods.prototype.runContextGraphSyncWithBackpressure.call( + agentLike as never, + createOperationContext('sync'), + 'urn:cg:private:e2e', + 'durable' as never, + 'durable:urn:cg:private:e2e', + () => new Promise((resolve) => { releaseWork = resolve; }), + { source: 'catchup-foreground' }, + ); + await tick(); + + // Release in `finally`: this admission is registered in the SHARED + // backpressureRegistry, so a failed assertion that skipped the release would + // leave it active and cascade into every later test in this file. + try { + const snapshot = backpressureRegistry.capture().schedulers.find( + (scheduler) => scheduler.scheduler === 'sync-global', + ); + expect(snapshot).toMatchObject({ + lanes: [expect.objectContaining({ + activeOperations: [expect.objectContaining({ operation: 'durable:catchup-foreground' })], + })], + }); + // …and the Context Graph id still never reaches node-wide diagnostics. + expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); + } finally { + releaseWork(); + await admitted; + } + }); + it('removes CG and peer correlation identifiers from node-wide pressure diagnostics', async () => { const ctx = createOperationContext('sync'); const policy = resolveSyncGlobalBackpressure({ @@ -228,6 +276,7 @@ describe('sync global backpressure', () => { policy, ctx, label: 'durable:urn:cg:private:peer-a', + source: 'catchup-foreground', }, async () => new Promise((resolve) => { releaseRunning = resolve; @@ -239,6 +288,7 @@ describe('sync global backpressure', () => { policy, ctx, label: 'swm-recovery:urn:cg:private:peer-b', + source: 'reconcile', }, async () => undefined, ); @@ -247,10 +297,14 @@ describe('sync global backpressure', () => { const snapshot = backpressureRegistry.capture().schedulers.find( (scheduler) => scheduler.scheduler === 'sync-global', ); + // The operation dimension pairs the collapsed work class with the bounded + // admission origin, so a saturated queue can be attributed to a trigger + // (issue #2006 had to reconstruct that from daemon logs) without any + // Context Graph or peer identifier reaching node-wide diagnostics. expect(snapshot).toMatchObject({ lanes: [expect.objectContaining({ - activeOperations: [expect.objectContaining({ operation: 'durable' })], - queuedOperations: [expect.objectContaining({ operation: 'swm-recovery' })], + activeOperations: [expect.objectContaining({ operation: 'durable:catchup-foreground' })], + queuedOperations: [expect.objectContaining({ operation: 'swm-recovery:reconcile' })], })], }); expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); @@ -261,6 +315,46 @@ describe('sync global backpressure', () => { await Promise.all([running, queued]); }); + it('clamps an unknown admission origin instead of widening the diagnostic label space', async () => { + const ctx = createOperationContext('sync'); + const policy = resolveSyncGlobalBackpressure({ + syncGlobalMaxInflight: 1, + syncGlobalQueueLimit: 1, + }); + let releaseRunning!: () => void; + // The union is compile-time only; a value crossing the worker RPC boundary + // (or an outright cast) must not be able to smuggle an identifier into a + // metric/log dimension or blow up its cardinality. + const running = withGlobalSyncBackpressure( + { + policy, + ctx, + label: 'durable:cg-x:peer-x', + // The option is typed `SyncAdmissionSource`; the cast is the point — + // the scheduler clamp is defence in depth for exactly this. + source: 'leak-urn:cg:private:xyz' as never, + }, + async () => new Promise((resolve) => { + releaseRunning = resolve; + }), + ); + await tick(); + + const snapshot = backpressureRegistry.capture().schedulers.find( + (scheduler) => scheduler.scheduler === 'sync-global', + ); + expect(snapshot).toMatchObject({ + lanes: [expect.objectContaining({ + activeOperations: [expect.objectContaining({ operation: 'durable:unspecified' })], + })], + }); + expect(JSON.stringify(snapshot)).not.toContain('leak-'); + expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); + + releaseRunning(); + await running; + }); + it('starts a later elevated CG before an earlier deprioritized queued CG', async () => { const ctx = createOperationContext('sync'); const policy = resolveSyncGlobalBackpressure({ syncGlobalMaxInflight: 1, syncGlobalQueueLimit: 3 }); diff --git a/packages/agent/test/sync-fetch-coalescing.test.ts b/packages/agent/test/sync-fetch-coalescing.test.ts index e9071f98af..beb7a8840b 100644 --- a/packages/agent/test/sync-fetch-coalescing.test.ts +++ b/packages/agent/test/sync-fetch-coalescing.test.ts @@ -790,6 +790,7 @@ describe('DKGAgent sync fetch coalescing', () => { const remotePeer = { toString: () => PEER_A }; const order: string[] = []; const priorities: Array = []; + const sources: Array = []; let durableCalls = 0; try { @@ -802,10 +803,11 @@ describe('DKGAgent sync fetch coalescing', () => { _onPhase: unknown, _onAccessDenied: unknown, _sinceBatchIdFor: unknown, - options: { priority?: number } | undefined, + options: { priority?: number; source?: string } | undefined, ) => { durableCalls += 1; priorities.push(options?.priority); + sources.push(options?.source); order.push(`durable-${durableCalls}`); return durableCalls === 1 ? { @@ -818,9 +820,10 @@ describe('DKGAgent sync fetch coalescing', () => { (agent as any).syncSharedMemoryFromPeerDetailed = async ( _peerId: string, _contextGraphIds: string[], - options: { priority?: number } | undefined, + options: { priority?: number; source?: string } | undefined, ) => { priorities.push(options?.priority); + sources.push(options?.source); order.push('shared'); return cleanSharedMemorySyncResult(); }; @@ -839,6 +842,15 @@ describe('DKGAgent sync fetch coalescing', () => { FOREGROUND_CATCHUP_SYNC_PRIORITY, FOREGROUND_CATCHUP_SYNC_PRIORITY, ]); + // The admission ORIGIN travels with the priority on the in-agent runner + // too: dropping it here while keeping the priority would silently report + // inline foreground catch-up as `durable:unspecified` in node-wide + // scheduler diagnostics (issue #2006). + expect(sources).toEqual([ + 'catchup-foreground', + 'catchup-foreground', + 'catchup-foreground', + ]); } finally { await agent.stop().catch(() => {}); } diff --git a/packages/agent/test/sync-on-connect-churn.test.ts b/packages/agent/test/sync-on-connect-churn.test.ts index 92803e51fd..ae3fbe1d31 100644 --- a/packages/agent/test/sync-on-connect-churn.test.ts +++ b/packages/agent/test/sync-on-connect-churn.test.ts @@ -118,6 +118,31 @@ describe('sync-on-connect churn gates', () => { expect(calls).toEqual([PEER_A]); }); + it('labels the connect-driven admission on-connect, not unspecified', async () => { + // The complement of the reconciler assertion below, and the source with the + // highest production volume: every reconnect sync flows through this + // default. Nothing pinned it, so changing the default would silently + // relabel most sync-global pressure on the operator dashboards. + const agent = await createUnstartedAgent('SyncOnConnectSourceLabel'); + (agent as any).started = true; + const sources: unknown[] = []; + (agent as any).trySyncFromPeer = async ( + _peer: string, + _onAccounting: unknown, + source: unknown, + ) => { + sources.push(source); + return undefined; + }; + + await (agent as any).attemptSyncFromPeerWithReconcilerAccounting(PEER_A, { + connected: true, + hasSyncProtocol: true, + }); + + expect(sources).toEqual(['on-connect']); + }); + it('reconciler still retries stale connected peers', async () => { const agent = await createUnstartedAgent('SyncReconcilerStillRetries'); (agent as any).started = true; @@ -132,7 +157,10 @@ describe('sync-on-connect churn gates', () => { await (agent as any).reconcileSyncFromConnectedPeers(); await flushTimers(); - expect(trySyncFromPeer.calls).toEqual([[PEER_A, expect.any(Function)]]); + // The third argument is the bounded admission origin (issue #2006): the + // reconciler's queue pressure must be attributable to `reconcile`, not + // indistinguishable from sync-on-connect. + expect(trySyncFromPeer.calls).toEqual([[PEER_A, expect.any(Function), 'reconcile']]); }); it('records backoff after a failed sync round and blocks connection-open rescheduling', async () => { diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index b43efe5065..190eb3878c 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -1,12 +1,16 @@ import { describe, expect, it } from 'vitest'; import { + SYNC_ADMISSION_SOURCES, contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, normalizeSyncContextGraphPriorities, orderContextGraphIdsByPriority, syncPriorityClass, validateSyncResponderSnapshotLimitsConfig, } from '../src/sync/policy.js'; +import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; +import { authoritativeSyncPeerId, resolveCuratorSyncPeer } from '../src/dkg-agent-cg-resolve.js'; describe('sync Context Graph policy', () => { it('normalizes safe integer priorities and preserves stable input order for ties', () => { @@ -56,3 +60,441 @@ describe('sync responder snapshot config validation', () => { .toThrow(`syncResponderSnapshotLimits.${path}`); }); }); + +describe('normalizeSyncAdmissionSource', () => { + it('passes through every declared admission origin', () => { + for (const source of SYNC_ADMISSION_SOURCES) { + expect(normalizeSyncAdmissionSource(source)).toBe(source); + } + }); + + it('clamps unknown, absent, and identifier-bearing origins to `unspecified`', () => { + // These values become metric and log dimensions on the node-wide + // `sync-global` scheduler, so the label space is a contract: an unbounded + // or identifier-bearing origin would re-open the correlation-identifier + // leak that collapsing the operation label was added to close, and would + // multiply the diagnostic cardinality. + expect(normalizeSyncAdmissionSource(undefined)).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('Catchup-Foreground')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('durable:urn:cg:private:abc')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('__proto__')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('toString')).toBe('unspecified'); + }); + + it('keeps the declared origin set small and free of punctuation', () => { + expect(new Set(SYNC_ADMISSION_SOURCES).size).toBe(SYNC_ADMISSION_SOURCES.length); + expect(SYNC_ADMISSION_SOURCES.length).toBeLessThanOrEqual(12); + for (const source of SYNC_ADMISSION_SOURCES) { + expect(source).toMatch(/^[a-z][a-z-]*$/); + } + }); +}); + +/** + * These drive the REAL resolver — `resolveCuratorSyncPeer` and the two + * lifecycle methods on their actual prototypes — not a stub of it. Which of + * the two routes produced a peer id is what decides whether one peer's answer + * may stand for a whole Context Graph, and that decision is made inside the + * resolver, so stubbing it out would leave the interesting half untested. + */ +describe('curator sync-peer provenance', () => { + const CG = 'cg/provenance'; + const HINT = '12D3KooWBootstrapHint'; + const CURATOR = '12D3KooWMetadataCurator'; + + type MetaFacts = { + curator?: string; + curators?: string[]; + creator?: string; + creators?: string[]; + declared?: boolean; + accessPolicy?: string; + }; + + /** + * Shape a fixture the way the projection actually shapes a record. + * + * `applyFact` records ONE declared curator twice — `pushUnique(record.curators, + * o)` AND `record.curator ??= o` — so a real single-curator graph arrives with + * `curators.length === 1` and a matching scalar, i.e. two entries once the + * scalar is prepended. Fixtures that set only the scalar and leave the array + * empty do NOT look like production, and a cardinality bug that rejects every + * real graph would pass against them. Deriving the arrays here keeps every + * fixture in the production shape by construction. + */ + function projected(facts: MetaFacts) { + return { + ...facts, + curators: facts.curators ?? (facts.curator ? [facts.curator] : []), + creators: facts.creators ?? (facts.creator ? [facts.creator] : []), + }; + } + + /** + * `getCgMeta` is the MERGED projection (`_meta` + AGENTS + `_catalog` + + * ONTOLOGY); `getOwnCgMetaFacts` is what the Context Graph declared about + * itself. `ownMeta` defaults to `meta` — the ordinary case where they agree — + * so any test exercising the difference has to say so out loud. + * + * `ownMeta` also carries a COMPLETE canonical definition by default + * (`declared` + an access policy), because that is what a receiver's `_meta` + * always holds: `curator-meta-refresh` refuses to install a snapshot without + * it. A test that wants the partial shape states it explicitly. + */ + function agentWithMeta( + meta: MetaFacts, + findAgents: () => Promise> = async () => [], + ownMeta: MetaFacts = meta, + ) { + return { + getCgMeta: async () => projected(meta), + getOwnCgMetaFacts: async () => projected({ + declared: true, + accessPolicy: 'private', + ...ownMeta, + }), + discovery: { findAgents }, + }; + } + + it('prefers a declared curator over the bootstrap hint, and consumes the hint', async () => { + // The ordinary case on a healthy network: the join approval came from the + // curator, so both routes name the same peer. The declared route wins and + // the hint is spent — but the result RANKS the walk, it does not end it. + const hints = new Map([[CG, CURATOR]]); + const agent = agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + expect(hints.has(CG)).toBe(false); + }); + + it('marks an echoed bootstrap hint as NOT authoritative', async () => { + // With no curator in `_meta` the resolver echoes the join-approval hint. + // That hint can be stale — peer ids are cryptographic identities, so a + // curator that rotated its libp2p key leaves an ordinary member on the id + // it still names — so it may rank the walk but must never end it. + const hints = new Map([[CG, HINT]]); + + expect(await resolveCuratorSyncPeer(agentWithMeta({}) as never, hints, CG)) + .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + // A non-DKG curator DID is equally unresolvable. + expect(await resolveCuratorSyncPeer(agentWithMeta({ curator: 'did:web:example' }) as never, hints, CG)) + .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + // …as is a wallet-address curator no registry can resolve. + expect(await resolveCuratorSyncPeer( + agentWithMeta({ curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }) as never, + hints, + CG, + )).toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + // The hint survives every fallback: it is still the best peer available. + expect(hints.get(CG)).toBe(HINT); + }); + + it('reports no peer when neither route produced one', async () => { + expect(await resolveCuratorSyncPeer(agentWithMeta({}) as never, new Map(), CG)) + .toEqual({ provenance: 'none' }); + }); + + it('never lets a registry match be an authority, however unique it looks locally', async () => { + // `findAgents()` queries the LOCAL Agent Registry only, so "one match" means + // one match on THIS node — not that the wallet has a single registration on + // the network. Local cardinality cannot establish a binding, so this route + // ranks and never settles. + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [{ agentAddress: '0x00000000000000000000000000000000000000AB', peerId: CURATOR }], + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('registry'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('resolves a wallet curator from a declared creator WITHOUT granting authority', async () => { + // The graph's own `/_meta` declaring both the curator DID and the + // creator peer is the strongest statement available locally — and it is + // still not enough. Ordinary durable-meta catch-up admits descriptive rows + // for the Context Graph's entity subject and writes them into that very + // graph, so a contacted peer can supply these exact rows. Reading one graph + // instead of the merged projection identifies the GRAPH, never the WRITER. + const hints = new Map([[CG, HINT]]); + const declared = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const agent = agentWithMeta(declared, async () => { + throw new Error('a declared creator must not need the registry'); + }, declared); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('demotes a creator the MERGED projection supplied but the graph did not', async () => { + // `getCgMeta()` unions `_meta` with AGENTS / `_catalog` / ONTOLOGY under + // first-wins precedence and discards which graph supplied each fact. A + // creator contributed by an AGENTS-only declaration therefore looks identical + // to one the graph declared about itself — but only the latter may end the + // walk. Here the projection offers a creator the graph's own `_meta` does not. + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => [], + // The graph names the curator, but binds no creator peer itself. + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('ranks but never settles a graph whose creator is not in its own _meta', async () => { + // The public shape: the merged projection resolves a creator peer (from the + // network-replicated ONTOLOGY graph), but the graph's own `_meta` declares + // no creator. That peer is still the best one to walk FIRST — it is + // returned as the resolved peer — but it may not end the walk on a claim + // that the graph is empty. The fan-out reduction for this shape comes from + // the peer's verified DATA narrowing the rest, which needs no trust; see + // `preferredEvidence` in `catchup-runner-worker-impl.ts`. + const agent = agentWithMeta( + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => { + throw new Error('the registry fallback must not be reached here'); + }, + // The graph itself names the curator but binds no creator peer. + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('refuses authority to a graph that never declared itself', async () => { + // The reviewer's "identity-only partial own _meta": a couple of rows landed + // in `/_meta` without the canonical definition. `_meta` accumulates from + // several writers, so a curator row alone does not mean this node holds the + // graph's definition — and two stray triples must not be able to end a walk. + const declared = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const agent = agentWithMeta(declared, async () => [], { + ...declared, + declared: false, + accessPolicy: undefined, + }); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('refuses authority when the graph declares two candidate creator peers', async () => { + // `createContextGraph` calls this shape out directly — "stray creator/curator + // triples (e.g. from a previous build that backfilled per node)" — and the + // merged projection picks among creators in arbitrary order. Accepting + // whichever one happens to match the candidate peer lets a stale member + // speak for the graph. + const agent = agentWithMeta( + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => { + throw new Error('the registry fallback must not be reached here'); + }, + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWStaleMember'], + }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('refuses authority when the graph declares two different curators', async () => { + const agent = agentWithMeta( + { curator: `did:dkg:agent:${CURATOR}` }, + async () => [], + { + curator: `did:dkg:agent:${CURATOR}`, + curators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWOtherCurator'], + }, + ); + + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(agent as never, new Map(), CG), + )).toBeUndefined(); + }); + + it('demotes when the graph does not name that curator at all', async () => { + // A curator the merged projection asserts but the graph never claimed. + const agent = agentWithMeta( + { curator: `did:dkg:agent:${CURATOR}` }, + async () => [], + {}, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('fails closed to ranking when the source-qualified read is unavailable', async () => { + // A receiver without the reader, or one whose read throws, must never be + // upgraded to authority. + const noReader = { + getCgMeta: async () => ({ curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }), + discovery: { findAgents: async () => [] }, + }; + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(noReader as never, new Map(), CG), + )).toBeUndefined(); + + const throwingReader = { + ...noReader, + getOwnCgMetaFacts: async () => { throw new Error('store unavailable'); }, + }; + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(throwingReader as never, new Map(), CG), + )).toBeUndefined(); + }); + + it('will not make an AMBIGUOUS registry match an authority', async () => { + // `findAgents()` returns whichever registrations exist for a wallet, and the + // code has always documented the pick as arbitrary when there are several. + // That was harmless while this only ranked the walk. It is not harmless now + // that `'metadata'` means "may end the walk": an ordinary member sharing the + // curator's wallet could answer with a clean subset and stop the walk before + // the real curator is ever contacted. + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [ + { agentAddress: '0x00000000000000000000000000000000000000AB', peerId: '12D3KooWMemberA' }, + { agentAddress: '0x00000000000000000000000000000000000000ab', peerId: '12D3KooWMemberB' }, + ], + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + // It still RANKS the walk — an arbitrary co-registrant is a better first try + // than nothing… + expect(resolved.peerId).toBe('12D3KooWMemberA'); + expect(resolved.provenance).toBe('registry'); + // …but it can never END it. + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('grants authority to NO route, however well the graph declares itself', async () => { + // The invariant, stated once over every shape that has ever been proposed + // as sufficient. Each of these ranks the walk correctly; none may end it, + // because none is attributable to a writer a peer cannot impersonate. + // + // If a future change introduces a real trust anchor — a curator-signed + // snapshot, an on-chain curator→peer edge, or the locally persisted + // join-approval record — this test is where the new positive case belongs, + // NOT a relaxation of the shapes below. + const declaredWallet = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const shapes = [ + // A bare peer-id DID curator: the DID IS the peer, nothing to reconcile. + agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }), + // A wallet curator with the creator peer declared alongside it. + agentWithMeta(declaredWallet, async () => [], declaredWallet), + // The same, resolved through the local Agent Registry instead. + agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [{ agentAddress: '0x00000000000000000000000000000000000000AB', peerId: CURATOR }], + ), + ]; + + for (const agent of shapes) { + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).not.toBe('metadata'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + } + }); + + it('answers ranking and authority from ONE resolution', async () => { + // The catch-up boundary needs both notions, and resolving twice is not + // free or even equivalent: each resolution reads `_meta` (and can drive the + // registry fallback), and the resolver EVICTS the bootstrap hint once + // metadata confirms a curator, so the second call runs against a different + // map than the first. + let metaReads = 0; + // Production shape: the projection records the declared curator both as the + // scalar and in the array, alongside the canonical definition facts. + const declared = { + declared: true, + accessPolicy: 'private', + curator: `did:dkg:agent:${CURATOR}`, + curators: [`did:dkg:agent:${CURATOR}`], + creators: [], + }; + const agent = { + preferredSyncPeers: new Map([[CG, CURATOR]]), + getCgMeta: async () => { + metaReads += 1; + return declared; + }, + // The graph declares this curator itself, so the binding is authoritative. + getOwnCgMetaFacts: async () => declared, + discovery: { findAgents: async () => [] }, + }; + + const resolved = await LifecycleSyncMethods.prototype.resolveSyncPeerWithProvenance + .call(agent as never, CG); + + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); + expect(metaReads).toBe(1); + // Both narrow notions are derivable from it, matching the wrappers exactly. + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + expect(authoritativeSyncPeerId({ peerId: CURATOR, provenance: 'bootstrap-hint' })) + .toBeUndefined(); + expect(authoritativeSyncPeerId({ provenance: 'none' })).toBeUndefined(); + }); + + it('ranks on any provenance and settles the walk on none', async () => { + // The two lifecycle entry points, on their real prototypes: ranking takes + // whatever peer is available; authority takes nothing at all. + const confirmedCurator = { + preferredSyncPeers: new Map([[CG, CURATOR]]), + ...agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }), + }; + const hintOnly = { + preferredSyncPeers: new Map([[CG, HINT]]), + ...agentWithMeta({}), + }; + const rank = LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId; + const authority = LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId; + + expect(await rank.call(confirmedCurator as never, CG)).toBe(CURATOR); + expect(await authority.call(confirmedCurator as never, CG)).toBeUndefined(); + + expect(await rank.call(hintOnly as never, CG)).toBe(HINT); + expect(await authority.call(hintOnly as never, CG)).toBeUndefined(); + }); +}); diff --git a/packages/agent/vitest.unit.config.ts b/packages/agent/vitest.unit.config.ts index f2842594bb..51776d5c88 100644 --- a/packages/agent/vitest.unit.config.ts +++ b/packages/agent/vitest.unit.config.ts @@ -79,6 +79,11 @@ export default defineConfig({ "test/sync-fetch-coalescing.test.ts", "test/sync-fetch-coalescing-durable.test.ts", "test/sync-backpressure.test.ts", + "test/sync-policy.test.ts", + "test/catchup-policy.test.ts", + "test/catchup-concurrency.test.ts", + "test/map-with-concurrency.test.ts", + "test/peer-selection.test.ts", "test/sync-requester-priority.test.ts", "test/sync-requester-progress.test.ts", "test/rootless-durable-bounded-progress.test.ts", diff --git a/packages/cli/src/api-client.ts b/packages/cli/src/api-client.ts index 6199a6eb28..c06e4a3e48 100644 --- a/packages/cli/src/api-client.ts +++ b/packages/cli/src/api-client.ts @@ -1500,6 +1500,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; @@ -1573,6 +1575,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; @@ -1642,6 +1646,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 988e151493..89fc45418d 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -1,15 +1,26 @@ import { parentPort } from 'node:worker_threads'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, + catchupWaveSizes, createFailedPeerDurableSyncResult, mapWithConcurrency, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, + type CatchupPlaneContext, + type DurableSyncResult, + type SharedMemorySyncResult, } from '@origintrail-official/dkg-agent'; import { + addCatchupPlaneEvidence, + catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByAuthorityHostedEmpty, + catchupPlaneProvenByData, type CatchupJobResult, + type CatchupPlaneCompletionEvidence, type CatchupRunRequest, } from './catchup-runner.js'; @@ -50,9 +61,60 @@ parentPort!.on('message', async (message: any) => { } }); +/** + * One peer's durable plane. The wire shape is the agent's own + * `DurableSyncResult`, structured-cloned back across the Worker RPC; + * `verifiedPrivateOnlyResponses` is normalized to a number on arrival so the + * accumulation below never has to re-guard it. + */ +type CatchupDurableResult = DurableSyncResult & { verifiedPrivateOnlyResponses: number }; + +/** One peer's shared-memory plane, as returned across the Worker RPC. */ +type CatchupSharedMemoryResult = SharedMemorySyncResult; + +/** + * One peer's sync round. A plane is `null` when the walk deliberately skipped + * it because the authority already settled that plane — that is the ONLY + * exceptional case, and it is distinct from a plane that ran and failed. + */ +interface PeerRound { + peerId: string; + /** The resolved curator for this Context Graph produced this round. */ + fromAuthority: boolean; + durable: CatchupDurableResult | null; + shared: CatchupSharedMemoryResult | null; +} + +function emptyShared(): CatchupSharedMemoryResult { + return { + insertedTriples: 0, + fetchedMetaTriples: 0, + fetchedDataTriples: 0, + insertedMetaTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 0, + checkpointAdvances: 0, + emptyResponses: 0, + droppedDataTriples: 0, + failedPeers: 1, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + async function runCatchup(request: CatchupRunRequest): Promise { const prepared = await invoke<{ preferredPeerId?: string; + /** + * The preferred peer ONLY when it came from authoritative Context Graph + * metadata. A join-approval bootstrap hint arrives as `preferredPeerId` + * without this, so it orders the walk but can never end it. + */ + authoritativePeerId?: string; isPrivateContextGraph: boolean; peerIds: string[]; connectedPeers: number; @@ -69,8 +131,13 @@ async function runCatchup(request: CatchupRunRequest): Promise let noProtocolPeers = 0; const cleanPlaneCompletions: NonNullable = { - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0 }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, }; const diagnostics: NonNullable = { @@ -114,20 +181,11 @@ async function runCatchup(request: CatchupRunRequest): Promise }, }; - // Run per-peer syncs in parallel, but BOUNDED. The sequential version here - // used to walk the peer set one at a time, which meant a curated-CG denial - // from a 10-peer pool took 10 × (syncDurable timeout + syncSharedMemory - // timeout) to report back — often minutes. Codex N18 then parallelised this - // Worker path (the daemon `/api/context-graph/subscribe` route) with an - // unbounded `Promise.all` — which made it the 2026-07-07 mainnet sync-storm - // engine: one subscribe on a high-degree node fired a full durable+SWM pull - // at EVERY sync-capable peer at once, saturating the triple store. Mirror - // the agent-side `syncContextGraphFromConnectedPeers` fix: run the fan-out - // through `mapWithConcurrency` under the shared cap - // (CATCHUP_MAX_CONCURRENT_PEER_SYNCS, env DKG_CATCHUP_MAX_CONCURRENT_PEERS) - // so both runners have the same latency AND the same load ceiling. The - // protocol probe below is lighter but fans out over the full post-prime-dial - // peer list, so it gets the same bound. + // Probe every connected peer for PROTOCOL_SYNC up front, bounded by the shared + // catch-up cap. This stays eager on purpose: `syncCapablePeers` and + // `noProtocolPeers` are read by daemon status mapping as counts over the whole + // connected set ("no sync-capable peers found — the curator may be offline"), + // and the probe is a peerStore lookup, not a transfer. const checked = await mapWithConcurrency( prepared.peerIds, CATCHUP_MAX_CONCURRENT_PEER_SYNCS, @@ -145,96 +203,165 @@ async function runCatchup(request: CatchupRunRequest): Promise syncCapable.push(peerId); } syncCapablePeers = syncCapable.length; - peersTried = syncCapable.length; - // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we can - // from the other peers instead of failing the entire subscribe/catch-up immediately. - const emptyShared = () => ({ - insertedTriples: 0, - fetchedMetaTriples: 0, - fetchedDataTriples: 0, - insertedMetaTriples: 0, - insertedDataTriples: 0, - bytesReceived: 0, - resumedPhases: 0, - timedOutPhases: 0, - completedPhases: 0, - checkpointAdvances: 0, - emptyResponses: 0, - droppedDataTriples: 0, - failedPeers: 1, - failedPhases: 0, - deniedPhases: 0, - deferredBackpressure: 0, - }); - // Bounded fan-out (sync-storm mitigation C-1): at most - // CATCHUP_MAX_CONCURRENT_PEER_SYNCS full per-peer sync rounds in flight. - // Every sync-capable peer is still synced and the result array is unchanged - // (input order, one entry per peer, per-peer failures isolated by the - // `.catch`es inside the callback) — the load is just staggered into waves. - const perPeerResults = await mapWithConcurrency( - syncCapable, - CATCHUP_MAX_CONCURRENT_PEER_SYNCS, - async (peerId) => { - return runCatchupPlanesWithPolicy({ - mode: 'foreground', - includeSharedMemory: request.includeSharedMemory, - syncDurable: async ({ priority }) => { - const rawDurable = await invoke( - 'syncDurable', - peerId, - request.contextGraphId, - priority, - ).catch(() => createFailedPeerDurableSyncResult()); - return { - ...rawDurable, - verifiedPrivateOnlyResponses: rawDurable.verifiedPrivateOnlyResponses ?? 0, - }; - }, - syncSharedMemory: ({ priority }) => invoke( - 'syncSharedMemory', - peerId, - request.contextGraphId, - priority, - ).catch(() => emptyShared()), - }); - }, - ); - for (const { durable, shared } of perPeerResults) { + // Only the resolved curator's snapshot is a reference for the WHOLE graph: a + // peer's `complete` flag proves it served its own manifest, so a + // non-authoritative peer carrying a subset would otherwise be able to cut the + // walk short and strand another peer's Knowledge Assets. Both optimisations + // below — skipping remaining peers, and skipping an already-proven plane on + // the peers we do contact — are therefore gated on AUTHORITY proof. With no + // resolvable curator the walk degrades to the previous full bounded fan-out + // and keeps unioning every peer's data. + const authorityProven = { durable: false, sharedMemory: false }; + const authorityProvedEverything = (): boolean => authorityProven.durable + && (!request.includeSharedMemory || authorityProven.sharedMemory); + + /** + * The CURATOR's own evidence, kept apart from the round total. + * + * The round total mixes in every peer, and only the curator may end the walk — + * so proof-by-data has to be read from this, not from + * `cleanPlaneCompletions`, or any peer's data would stop it. + */ + const authorityEvidence: Record<'durable' | 'sharedMemory', CatchupPlaneCompletionEvidence> = { + durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, + sharedMemory: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, + }; + + /** + * Whether the curator's round settles a plane well enough to stop walking. + * + * Verified content from the curator always does. Its EMPTINESS is weaker: it + * is exactly `catchupPlaneProvenByAuthorityHostedEmpty`, the same predicate + * the readiness classifier applies — called here with the round's diagnostics + * so the two cannot disagree. + * + * That matters, because the round can contradict the curator. If another peer + * fetched data, or served content that failed verification, the curator's + * "there is nothing here" is stale and readiness voids it. Stopping the walk + * on it anyway would skip peers that might have delivered valid content and + * then report the job unready — the worst of both. + * + * Evaluated at the END of a wave rather than per peer, so a contradiction + * raised by ANY member of the same wave is already visible regardless of the + * order results happened to arrive in. + */ + const authoritySettles = ( + plane: 'durable' | 'sharedMemory', + ): boolean => catchupPlaneProvenByData(authorityEvidence[plane]) + || catchupPlaneProvenByAuthorityHostedEmpty( + authorityEvidence[plane], + diagnostics[plane], + { isPrivate: prepared.isPrivateContextGraph }, + ); + + /** + * Did the curator cleanly answer this plane at all? + * + * Separate from `authorityProven`: a curator that answered with data proves the + * plane, and one that answered content-free may or may not, but BOTH count as + * having answered. What readiness needs to know is the third case — the curator + * was selected and we never heard a clean word from it — because then a + * stranger's empty response cannot stand for the graph. + */ + const authorityAnswered = { durable: false, sharedMemory: false }; + + /** Fold the wave's accumulated state into the stop flags. */ + const settleAuthorityForWave = (): void => { + if (!authorityProven.durable && authoritySettles('durable')) { + authorityProven.durable = true; + } + if (request.includeSharedMemory + && !authorityProven.sharedMemory + && authoritySettles('sharedMemory')) { + authorityProven.sharedMemory = true; + } + }; + + // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we + // can from the other peers instead of failing the entire subscribe/catch-up. + const syncPeer = async (peerId: string): Promise => { + const syncDurable = ( + { priority, source }: CatchupPlaneContext, + ): Promise => + invoke('syncDurable', peerId, request.contextGraphId, priority, source) + .catch(() => createFailedPeerDurableSyncResult()) + .then((rawDurable) => ({ + ...rawDurable, + verifiedPrivateOnlyResponses: rawDurable.verifiedPrivateOnlyResponses ?? 0, + })); + const syncSharedMemory = ( + { priority, source }: CatchupPlaneContext, + ): Promise => + invoke('syncSharedMemory', peerId, request.contextGraphId, priority, source) + .catch(() => emptyShared()); + + // Narrow each fallback peer to the planes the AUTHORITY has not already + // settled. One plane is often settled long before the other — a Context + // Graph whose public VM data is empty can never prove its durable plane by + // data — so without this a single unproven plane would drag a full re-pull + // of the already-settled plane out of every remaining peer, which is the + // amplification this fix exists to remove. The kill-switch restores the + // previous fan-out faithfully: every peer, both requested planes. + const optimize = CATCHUP_STOP_ON_PROOF; + const needDurable = !optimize || !authorityProven.durable; + const needSharedMemory = request.includeSharedMemory + && (!optimize || !authorityProven.sharedMemory); + const fromAuthority = prepared.authoritativePeerId !== undefined + && peerId === prepared.authoritativePeerId; + if (!needDurable) { + const shared = needSharedMemory + ? await runCatchupPlaneWithPolicy('foreground', syncSharedMemory) + : null; + return { peerId, fromAuthority, durable: null, shared }; + } + const round = await runCatchupPlanesWithPolicy({ + mode: 'foreground', + includeSharedMemory: needSharedMemory, + syncDurable, + syncSharedMemory, + }); + return { peerId, fromAuthority, ...round }; + }; + + const accumulate = ({ durable, shared, fromAuthority }: PeerRound): void => { let peerDenied = false; - dataSynced += durable.insertedDataTriples ?? 0; - diagnostics.durable.fetchedMetaTriples += durable.fetchedMetaTriples; - diagnostics.durable.fetchedDataTriples += durable.fetchedDataTriples; - diagnostics.durable.insertedMetaTriples += durable.insertedMetaTriples; - diagnostics.durable.insertedDataTriples += durable.insertedDataTriples; - diagnostics.durable.bytesReceived += durable.bytesReceived; - diagnostics.durable.resumedPhases += durable.resumedPhases; - diagnostics.durable.timedOutPhases += durable.timedOutPhases ?? 0; - diagnostics.durable.completedPhases += durable.completedPhases ?? 0; - diagnostics.durable.checkpointAdvances += durable.checkpointAdvances ?? 0; - diagnostics.durable.emptyResponses += durable.emptyResponses; - diagnostics.durable.metaOnlyResponses += durable.metaOnlyResponses; - diagnostics.durable.verifiedPrivateOnlyResponses += - durable.verifiedPrivateOnlyResponses; - diagnostics.durable.dataRejectedMissingMeta += durable.dataRejectedMissingMeta; - diagnostics.durable.rejectedKcs += durable.rejectedKcs; - diagnostics.durable.failedPeers += durable.failedPeers; - diagnostics.durable.failedPhases += durable.failedPhases ?? 0; - diagnostics.durable.deferredBackpressure += durable.deferredBackpressure ?? 0; - deferredBackpressure += durable.deferredBackpressure ?? 0; - diagnostics.durable.deniedPhases = - (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); - peerDenied = peerDenied || durable.deniedPhases > 0; - - if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { - if ((durable.insertedDataTriples ?? 0) > 0) { - cleanPlaneCompletions.durable.verifiedDataPeers += 1; - } - if (durable.verifiedPrivateOnlyResponses > 0) { - cleanPlaneCompletions.durable.verifiedPrivateOnlyPeers += 1; - } - if ((durable.emptyResponses ?? 0) > 0) { - cleanPlaneCompletions.durable.emptyPeers += 1; + if (durable) { + dataSynced += durable.insertedDataTriples ?? 0; + diagnostics.durable.fetchedMetaTriples += durable.fetchedMetaTriples; + diagnostics.durable.fetchedDataTriples += durable.fetchedDataTriples; + diagnostics.durable.insertedMetaTriples += durable.insertedMetaTriples; + diagnostics.durable.insertedDataTriples += durable.insertedDataTriples; + diagnostics.durable.bytesReceived += durable.bytesReceived; + diagnostics.durable.resumedPhases += durable.resumedPhases; + diagnostics.durable.timedOutPhases += durable.timedOutPhases ?? 0; + diagnostics.durable.completedPhases += durable.completedPhases ?? 0; + diagnostics.durable.checkpointAdvances += durable.checkpointAdvances ?? 0; + diagnostics.durable.emptyResponses += durable.emptyResponses; + diagnostics.durable.metaOnlyResponses += durable.metaOnlyResponses; + diagnostics.durable.verifiedPrivateOnlyResponses += + durable.verifiedPrivateOnlyResponses; + diagnostics.durable.dataRejectedMissingMeta += durable.dataRejectedMissingMeta; + diagnostics.durable.rejectedKcs += durable.rejectedKcs; + diagnostics.durable.failedPeers += durable.failedPeers; + diagnostics.durable.failedPhases += durable.failedPhases ?? 0; + diagnostics.durable.deferredBackpressure += durable.deferredBackpressure ?? 0; + deferredBackpressure += durable.deferredBackpressure ?? 0; + diagnostics.durable.deniedPhases = + (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); + peerDenied = peerDenied || durable.deniedPhases > 0; + + const durableEvidence = catchupPeerPlaneEvidence(durable, { + complete: durable.complete, + fromAuthority, + plane: 'durable', + }); + addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); + if (fromAuthority) { + addCatchupPlaneEvidence(authorityEvidence.durable, durableEvidence); + if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { + authorityAnswered.durable = true; + } } } @@ -259,12 +386,14 @@ async function runCatchup(request: CatchupRunRequest): Promise (diagnostics.sharedMemory.deniedPhases ?? 0) + (shared.deniedPhases ?? 0); peerDenied = peerDenied || shared.deniedPhases > 0; - if (catchupPlaneCompletedWithoutFailure(shared)) { - if ((shared.insertedDataTriples ?? 0) > 0) { - cleanPlaneCompletions.sharedMemory.verifiedDataPeers += 1; - } - if ((shared.emptyResponses ?? 0) > 0) { - cleanPlaneCompletions.sharedMemory.emptyPeers += 1; + // Shared memory carries no verified-private-only signal, so the shared + // evidence only ever has data/empty set — the same reducer still applies. + const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority, plane: 'shared-memory' }); + addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); + if (fromAuthority) { + addCatchupPlaneEvidence(authorityEvidence.sharedMemory, sharedEvidence); + if (catchupPlaneCompletedWithoutFailure(shared)) { + authorityAnswered.sharedMemory = true; } } } @@ -282,9 +411,93 @@ async function runCatchup(request: CatchupRunRequest): Promise // completed with no timeout. Mirrors the inline // `syncContextGraphFromConnectedPeers` path so both runners report the // same shape. - if (catchupPeerSucceeded(durable, shared, peerDenied, durable.complete)) { + if (catchupPeerSucceeded(durable, shared, peerDenied, durable?.complete)) { peersSucceeded += 1; } + }; + + // Progressive peer walk (issue #2006). The peer list arrives ranked + // authority-first (preferred/curator, then known cores, then the rest), but + // that ordering used never to become *selection*: every sync-capable peer got + // a full durable+SWM pull, so a 14-peer testnet downloaded the same graph 5-6 + // times (147,246 fetched triples for a 24,541-triple graph, ~278MB) and the + // node-wide sync-global queue (2 inflight / 4 queued) saturated against + // itself. + // + // Instead, walk escalating waves and stop as soon as the AUTHORITY has proven + // every requested plane with verified data — see `authorityProven` above for + // why only the curator's snapshot may cut the walk short. Wave 1 is that + // curator when one is resolvable, so the happy path transfers exactly one + // payload; with no resolvable curator nothing is ever authority-proven and + // this degrades to the previous full bounded fan-out. + // + // The stop condition is also deliberately POSITIVE-only: an empty round proves + // nothing on its own (an unrelated peer and an empty host are byte-identical + // on the wire), so emptiness stays a whole-round verdict evaluated by + // `catchupPlaneProvenByUnanimousEmpty` after every peer has been walked. + // + // Tradeoff, stated deliberately: even the curator's `complete` flag proves it + // served its own manifest, not that the manifest was network-complete. + // Foreground catch-up therefore optimises for one fast authoritative payload; + // breadth and eventual convergence remain the background reconcile lane's job. + // DKG_CATCHUP_STOP_ON_PROOF=0 restores the previous full fan-out. + // + // The single-peer opening wave is spent on the authority, so it is only taken + // when there IS one: if no curator resolved (or it is not sync-capable), the + // head of the ranked list has no special claim and serialising it would just + // add a round-trip to the front of every round, with no early stop to earn it + // back. In that case the walk opens at the full concurrency cap — the previous + // first-round latency. + // The `!== undefined` half is DEFENSIVE, not behavioural, and is called out + // as such so it does not read as a load-bearing clause a test should pin: + // with no resolvable curator and no sync-capable peers both sides are + // `undefined` and would compare equal, but `catchupWaveSizes(0, …)` is `[]` + // (pinned in `catchup-concurrency.test.ts`), so a zero-peer walk runs no + // waves and the opening width is unobservable. The comparison is what + // actually decides: narrow the opening wave only when the authority IS the + // peer that wave would contact. + const authorityFirst = prepared.authoritativePeerId !== undefined + && syncCapable[0] === prepared.authoritativePeerId; + // Waves exist ONLY so an authority can cut the walk short. With no authority + // resolvable nothing can ever break the loop, so splitting the peer set into + // waves cannot save a single fetch — it only adds a barrier between them, + // making the round SLOWER than the single bounded pass it replaced. Fall back + // to that pass rather than paying for a stop that cannot happen. + const canStopEarly = CATCHUP_STOP_ON_PROOF && prepared.authoritativePeerId !== undefined; + const waveSizes = canStopEarly + ? catchupWaveSizes( + syncCapable.length, + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + authorityFirst ? 1 : CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + ) + : [syncCapable.length]; + let cursor = 0; + for (const waveSize of waveSizes) { + const wave = syncCapable.slice(cursor, cursor + waveSize); + if (wave.length === 0) break; + cursor += wave.length; + peersTried += wave.length; + // Never cancel a wave member mid-stream: an aborted resumed session is + // indistinguishable from a responder supersede. Let the wave finish, then + // stop before dispatching the next one. + const rounds = await mapWithConcurrency( + wave, + Math.min(CATCHUP_MAX_CONCURRENT_PEER_SYNCS, wave.length), + syncPeer, + ); + for (const round of rounds) accumulate(round); + settleAuthorityForWave(); + if (CATCHUP_STOP_ON_PROOF && authorityProvedEverything()) break; + } + + // A curator we resolved but never heard cleanly from makes the round + // incomplete rather than empty. Recorded per plane, since a curator can answer + // one and fail the other. + if (prepared.authoritativePeerId !== undefined) { + diagnostics.durable.authorityUnanswered = !authorityAnswered.durable; + if (request.includeSharedMemory) { + diagnostics.sharedMemory.authorityUnanswered = !authorityAnswered.sharedMemory; + } } diagnostics.noProtocolPeers = noProtocolPeers; @@ -300,6 +513,7 @@ async function runCatchup(request: CatchupRunRequest): Promise peersTried, peersResponded, peersSucceeded, + peersNotAttempted: syncCapable.length - peersTried, deferredBackpressure, dataSynced, sharedMemorySynced, diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 2aa79a8b22..8c06daf29c 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -2,12 +2,15 @@ import { Worker } from 'node:worker_threads'; import { existsSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; import { + authoritativeSyncPeerId, classifyDurableProgress, normalizeDurableSyncResult, + normalizeSyncAdmissionSource, type DKGAgent, type DurableProgressSummary, type DurableSyncDiagnostics, type DurableSyncResult, + type SyncPeerResolution, } from '@origintrail-official/dkg-agent'; import { PROTOCOL_SYNC } from '@origintrail-official/dkg-core'; @@ -37,6 +40,13 @@ export interface CatchupJobResult { * progress or a clean non-metadata-only empty completion. */ peersSucceeded: number; + /** + * Sync-capable peers this run deliberately never contacted because an earlier + * wave already proved every requested plane. These are neither failures nor + * successes; they exist so status mapping and operators can tell an + * early-stopped run from a run where peers were unreachable. + */ + peersNotAttempted?: number; /** Context Graph phases deferred by this node's local sync scheduler. */ deferredBackpressure: number; dataSynced: number; @@ -50,16 +60,9 @@ export interface CatchupJobResult { * the same plane cleanly and stored verified data. */ cleanPlaneCompletions?: { - durable: { - verifiedDataPeers: number; - /** Peers that cleanly verified one or more V2 KAs with no public triples. */ - verifiedPrivateOnlyPeers: number; - emptyPeers: number; - }; - sharedMemory: { - verifiedDataPeers: number; - emptyPeers: number; - }; + /** Always carries `verifiedPrivateOnlyPeers`; only the durable plane can produce it. */ + durable: CatchupPlaneCompletionEvidence & { verifiedPrivateOnlyPeers: number }; + sharedMemory: CatchupPlaneCompletionEvidence; }; diagnostics?: { noProtocolPeers: number; @@ -83,6 +86,9 @@ export interface CatchupJobResult { failedPhases: number; deferredBackpressure: number; deniedPhases?: number; + /** A resolvable curator never cleanly answered this plane; see + * `catchupPlaneProvenByUnanimousEmpty`. */ + authorityUnanswered?: boolean; }; sharedMemory: { fetchedMetaTriples: number; @@ -100,6 +106,9 @@ export interface CatchupJobResult { failedPhases: number; deferredBackpressure: number; deniedPhases?: number; + /** A resolvable curator never cleanly answered this plane; see + * `catchupPlaneProvenByUnanimousEmpty`. */ + authorityUnanswered?: boolean; }; }; } @@ -504,8 +513,354 @@ export function catchupPlaneCompletedWithoutFailure( return classifyDurableProgress(progress, { complete }).completedWithoutFailure; } +/** Per-plane clean-completion evidence accumulated across the peers this run contacted. */ +export interface CatchupPlaneCompletionEvidence { + verifiedDataPeers: number; + /** Peers that cleanly verified one or more V2 KAs with no public triples. */ + verifiedPrivateOnlyPeers?: number; + emptyPeers: number; + /** + * The metadata-resolved curator cleanly completed this plane while hosting + * the graph and carrying no data at all. See + * {@link catchupPlaneProvenByUnanimousEmpty}. + */ + authorityEmptyPeers?: number; + /** + * Peers that ANSWERED this plane but whose round did not complete cleanly. + * + * Every other field here records what a peer proved. This one records what a + * peer left unresolved, and it exists because the absence of a peer from the + * positive counters is ambiguous: `catchupPeerPlaneEvidence` returns an + * all-zero record for an incomplete round, so a peer that answered EMPTY but + * did not finish paging is indistinguishable from a peer that was never + * contacted. That ambiguity is invisible to the round diagnostics too — an + * explicit `complete: false` is not a transport failure, so it never reaches + * `failedPeers`. + * + * Without it, a round of one clean-empty peer plus one incomplete-empty peer + * reads as unanimously empty. Pure transport failures are deliberately NOT + * counted here: an unreachable stranger is already `failedPeers`, and folding + * it in would pin legitimately empty graphs in a retry loop on a lossy + * network. + */ + incompleteResponders?: number; +} + +/** The aggregate per-plane counters a whole-round verdict is allowed to consult. */ +export interface CatchupPlaneRoundDiagnostics { + fetchedMetaTriples?: number; + fetchedDataTriples?: number; + emptyResponses?: number; + /** Peers that returned `_meta` and no data; durable-only. */ + metaOnlyResponses?: number; + failedPeers?: number; + failedPhases?: number; + timedOutPhases?: number; + deniedPhases?: number; + deferredBackpressure?: number; + /** Durable-only integrity rejections; the shared-memory plane never sets them. */ + dataRejectedMissingMeta?: number; + rejectedKcs?: number; + /** + * A metadata-resolved curator WAS selected for this walk and did not cleanly + * answer this plane — it transport-failed, timed out, was denied, or never got + * contacted. Distinct from `failedPeers`, which counts any unreachable peer. + */ + authorityUnanswered?: boolean; +} + +/** + * Reduce ONE peer's plane result to the evidence a round accumulates from it. + * + * This is the single definition of what a peer's round contributes, so the + * walk's stop condition and the readiness classifier cannot drift: the walk + * feeds one peer's evidence to {@link catchupPlaneProvenByData}, and readiness + * feeds the summed evidence to the same predicate. Adding a new verified-content + * signal therefore has exactly one place to change. + * + * A plane that did not complete cleanly contributes nothing at all. + */ +export function catchupPeerPlaneEvidence( + plane: + | (CatchupPhaseProgress & { emptyResponses?: number; fetchedDataTriples?: number }) + | null + | undefined, + options: { + /** + * Which plane this result came from. REQUIRED, and deliberately not + * defaulted: the strongest thing this function can say — hosted-empty + * evidence — is true on the durable plane and false on shared memory, so a + * defaulted `plane` would let a shared-memory call site silently take the + * durable branch. Only a test would notice, and the whole point is that a + * mistake here settles a plane nobody proved. + */ + plane: 'durable' | 'shared-memory'; + /** Durable-only lifecycle state; the shared plane has no `complete` concept. */ + complete?: boolean; + fromAuthority?: boolean; + }, +): CatchupPlaneCompletionEvidence { + const none = { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + }; + if (!plane) return none; + if (!catchupPlaneCompletedWithoutFailure(plane, options.complete)) { + // The peer answered and its round was not clean. A pure transport failure is + // NOT that: we never heard from it, it is already counted in `failedPeers`, + // and treating unreachable strangers as unresolved evidence would stop a + // genuinely empty graph from ever settling on a lossy network. + const answeredButUnresolved = (plane.failedPeers ?? 0) === 0; + return answeredButUnresolved ? { ...none, incompleteResponders: 1 } : none; + } + // "The host says there is nothing here." Only the curator can say it: a + // response is content-free either by being wire-empty or by carrying nothing + // but `_meta`, and only the metadata-resolved curator's silence about data + // means the graph has none. Any other peer's identical answer just means that + // peer does not have it. + const carriedNoData = (plane.insertedDataTriples ?? 0) === 0 + && (plane.fetchedDataTriples ?? 0) === 0; + // Whose emptiness counts, and on which plane. + // + // DURABLE: the Context Graph is the curator's. `/_meta` carries its own + // definition triples, so a curator serving them proves it hosts the graph, and + // a curator with no data means the graph has none. Both wire-empty and + // metadata-only rounds are hosted-empty evidence there. + // + // SHARED MEMORY: nobody's emptiness counts, not even the curator's. SWM is a + // per-agent-address layered union (`//`) contributed by many + // members, so a curator holding no SWM rows says nothing about the members' + // layers — it does not own them. Letting it settle the plane skipped peers that + // held valid rows and could report `sharedMemoryVerified` with + // `sharedMemorySynced: 0`. An empty SWM plane is still provable, but only as a + // WHOLE-ROUND verdict once every peer has answered, which is what + // `catchupPlaneProvenByUnanimousEmpty` is for. + // + // Verified DATA from the curator still settles either plane. That is the + // tradeoff this PR states openly — a peer's `complete` flag proves only its own + // manifest, and the background reconciler remains the convergence mechanism — + // and it is what keeps the amplification fixed for an SWM-heavy graph, which + // issue #2006 measured at 122,705 fetched triples on the shared plane alone. + const answered = options.plane !== 'shared-memory' + && ((plane.emptyResponses ?? 0) > 0 + || (plane.metaOnlyResponses ?? 0) > 0 + || (plane.insertedMetaTriples ?? 0) > 0); + return { + verifiedDataPeers: (plane.insertedDataTriples ?? 0) > 0 ? 1 : 0, + verifiedPrivateOnlyPeers: (plane.verifiedPrivateOnlyResponses ?? 0) > 0 ? 1 : 0, + emptyPeers: (plane.emptyResponses ?? 0) > 0 ? 1 : 0, + authorityEmptyPeers: options.fromAuthority && carriedNoData && answered ? 1 : 0, + }; +} + +/** Fold one peer's evidence into the running per-plane totals. */ +export function addCatchupPlaneEvidence( + total: CatchupPlaneCompletionEvidence, + peer: CatchupPlaneCompletionEvidence, +): void { + total.verifiedDataPeers += peer.verifiedDataPeers; + if (peer.verifiedPrivateOnlyPeers) { + total.verifiedPrivateOnlyPeers = (total.verifiedPrivateOnlyPeers ?? 0) + + peer.verifiedPrivateOnlyPeers; + } + total.emptyPeers += peer.emptyPeers; + if (peer.authorityEmptyPeers) { + total.authorityEmptyPeers = (total.authorityEmptyPeers ?? 0) + peer.authorityEmptyPeers; + } + if (peer.incompleteResponders) { + total.incompleteResponders = (total.incompleteResponders ?? 0) + peer.incompleteResponders; + } +} + +/** + * Positive proof: some peer cleanly completed this plane while carrying + * cryptographically verified content. This is the only evidence strong enough + * to stop contacting further peers mid-run, because it is the only evidence a + * single peer can produce on its own. + */ +export function catchupPlaneProvenByData( + completion: CatchupPlaneCompletionEvidence | undefined, +): boolean { + return (completion?.verifiedDataPeers ?? 0) > 0 + || (completion?.verifiedPrivateOnlyPeers ?? 0) > 0; +} + +/** + * Does any signal in this round rule out an empty verdict outright? + * + * Shared by BOTH empty-proof modes below, because these are not "a peer that + * failed" — they are evidence about the graph's contents that no peer's silence + * can outrank. + */ +function emptyVerdictContradicted( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, +): boolean { + // Verified content, obviously. + if (catchupPlaneProvenByData(completion)) return true; + // Data that arrived and was rejected. A peer SERVED CONTENT for this graph + // which then failed verification, so content exists even though we could not + // keep it. `classifyDurableProgress` already treats these as blocking + // failures per peer; this is the same rule applied to the round. + if ((diagnostics?.dataRejectedMissingMeta ?? 0) > 0 + || (diagnostics?.rejectedKcs ?? 0) > 0) return true; + // Data fetched anywhere in the round, whoever fetched it. + return (diagnostics?.fetchedDataTriples ?? 0) > 0; +} + +/** + * Proof mode 1 — the CURATOR hosts the graph and it holds nothing. + * + * A registered public graph that really is empty still carries definition + * triples in its own `/_meta`, so the peer hosting it answers + * metadata-only, never wire-empty, and could never satisfy the whole-round rule + * below. Its curator saying so is the only evidence such a graph can produce. + * + * Scoped to the metadata-resolved curator and nothing else. Any OTHER peer's + * metadata-only round is the commonest state on the network — a member that has + * `_meta` but has not synced the data yet — and accepting it would resettle + * issue #2006's exact failure as `done` with zero Knowledge Assets. + * + * Another peer merely failing part-way cannot contradict the curator; another + * peer producing CONTENT can, and that is what {@link emptyVerdictContradicted} + * checks — it means the curator's view is behind the network's. + */ +export function catchupPlaneProvenByAuthorityHostedEmpty( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + // Private planes stay proof-by-content only: an authorized-but-filtered + // response is indistinguishable from an empty one on this side of the wire. + if (options.isPrivate) return false; + if ((completion?.authorityEmptyPeers ?? 0) === 0) return false; + return !emptyVerdictContradicted(completion, diagnostics); +} + +/** + * Proof mode 2 — a whole round in which nobody had anything. + * + * A peer that has never heard of a Context Graph and a peer that hosts an empty + * one are byte-identical on the wire: an unknown CG has no access policy, so the + * responder authorizes the request and its CG-scoped queries simply return zero + * rows. The requester only reports `emptyResponses` when BOTH phase payloads are + * empty (`sync-verify-worker-impl.ts`), so an empty response can never carry + * hosting evidence — there is no per-peer signal that could distinguish the two. + * + * Emptiness is therefore a verdict over the whole round: some peer completed + * cleanly empty, nobody delivered any graph CONTENT, and no peer engaged and + * then failed part-way. That exact shape — 122,705 data triples fetched and + * five failed phases, with five unrelated peers answering empty — is what + * settled issue #2006's run as `done` with 1 KA out of 40, and either clause + * kills it on its own. + * + * `metaOnlyResponses` also kills it. A non-curator that returned `_meta` and no + * data is the ambiguous case this rule cannot resolve — the requester itself + * logs "peer may have empty or pruned data graph" — and without the curator + * present there is nothing to resolve it against. When the curator IS present, + * proof mode 1 has already settled the plane, so voiding here costs the + * legitimately-empty graph nothing. + * + * The verdict IS voided when the round had a resolvable curator that never + * cleanly answered (`authorityUnanswered`). The peer best placed to know is the + * one we failed to hear from, so "nobody had anything" is not established — the + * round is incomplete, not empty. That closes issue #2006's own symptom in its + * sharpest form: the walk puts a resolvable curator alone in wave 1, so when the + * curator transport-fails the walk moves on to strangers, one answers empty, and + * 40 Knowledge Assets get reported as zero. + * + * Scoped to the AUTHORITY rather than to `failedPeers`, and the difference is + * load-bearing. `failedPeers` counts any unreachable peer, so voiding on it would + * also kill the verdict when NO curator is resolvable at all — the state where + * the hosted-empty backstop structurally cannot fire — leaving a legitimately + * empty public graph pinned at `unreachable` by a single unreachable stranger. + * That is the liveness failure this rule was originally written to avoid, and it + * is still worth avoiding; it is only the curator's silence that is decisive. + * + * Two counters are deliberately NOT consulted: + * + * - `failedPeers`. A transport failure to a peer we never heard from, which on a + * live testnet can be most of the connected set. An unreachable STRANGER is + * evidence of nothing; an unreachable CURATOR is, and has its own signal above. + * - `fetchedMetaTriples`. A raw triple count, not a per-peer verdict: a delta + * sync legitimately carries the whole metadata phase with nothing newer than + * the watermark, and the requester deliberately does NOT flag that as + * metadata-only. Voiding on the raw count would make a legitimately empty + * public graph permanently unreadable rather than merely unproven. + * - `failedPeers`. That is a transport failure to a peer we never heard from — + * on a live testnet a majority of connected peers can be unreachable — and an + * unreachable stranger is evidence of nothing. A peer that DID engage and + * then failed shows up in `failedPhases` / `timedOutPhases` / `deniedPhases` + * / `deferredBackpressure`, all of which do void the verdict. + * + * Residual, unchanged from before this rule existed: if the only host is + * unreachable while another peer answers cleanly empty, the round still reads + * as empty. Readiness is re-derived on the next catch-up. + */ +export function catchupPlaneProvenByUnanimousEmpty( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + // Empty or metadata-only responses have never been able to prove that a + // private graph is fully synchronized; that stays unchanged. + if (options.isPrivate) return false; + if (emptyVerdictContradicted(completion, diagnostics)) return false; + // A non-curator that has `_meta` and no data cannot tell "the graph is empty" + // from "I have not synced it yet". See the note above. + if ((diagnostics?.metaOnlyResponses ?? 0) > 0) return false; + // A peer that answered but did not finish paging leaves the round unresolved: + // "nobody had anything" cannot be concluded while somebody's answer is still + // half-delivered. This is not covered by the phase counters below — an + // explicit `complete: false` is neither a failure nor a timeout — and it is + // the one shape that survives `catchupPeerPlaneEvidence` erasing the peer to + // an all-zero record. + if ((completion?.incompleteResponders ?? 0) > 0) return false; + // Completion evidence is PER-PEER and says explicitly whether that peer's + // round was clean; `diagnostics.emptyResponses` is a raw aggregate that counts + // an empty payload even when the peer's round was NOT complete. Where the + // runner supplied completion evidence it is the whole truth for this plane, so + // the aggregate must not re-admit a response the per-peer view already + // excluded — otherwise an explicitly incomplete empty result proves the plane + // ready. The aggregate is a fallback for legacy callers that carry no + // completion evidence at all, never a second chance for callers that do. + const cleanEmptyObserved = completion !== undefined + ? (completion.emptyPeers ?? 0) > 0 + : (diagnostics?.emptyResponses ?? 0) > 0; + if (!cleanEmptyObserved) return false; + // The one peer whose silence is decisive. See the note above for why this is + // scoped to the curator rather than to `failedPeers`. + if (diagnostics?.authorityUnanswered) return false; + return (diagnostics?.failedPhases ?? 0) === 0 + && (diagnostics?.timedOutPhases ?? 0) === 0 + && (diagnostics?.deniedPhases ?? 0) === 0 + && (diagnostics?.deferredBackpressure ?? 0) === 0; +} + +/** + * Canonical readiness proof for one catch-up plane: verified content, the + * curator's hosted-empty word, or a whole round in which nobody had anything — + * in that order of strength. + * + * The peer walk stops early only on {@link catchupPlaneProvenByData} or the + * curator's own round, so whenever this falls through to the unanimous-empty + * branch the full peer set really was walked and the "nobody saw anything" + * denominator is meaningful. + */ +export function catchupPlaneReady( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + return catchupPlaneProvenByData(completion) + || catchupPlaneProvenByAuthorityHostedEmpty(completion, diagnostics, options) + || catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options); +} + export function catchupPeerSucceeded( - durable: CatchupPhaseProgress, + durable: CatchupPhaseProgress | null | undefined, shared: CatchupPhaseProgress | null | undefined, peerDenied: boolean, durableComplete?: boolean, @@ -533,10 +888,13 @@ export function catchupPeerSucceeded( } export function catchupPeerResponded( - durable: CatchupPhaseProgress, + durable: CatchupPhaseProgress | null | undefined, shared: CatchupPhaseProgress | null | undefined, ): boolean { - const phaseResponded = (phase: CatchupPhaseProgress): boolean => { + // A plane the walk deliberately skipped (already proven by an earlier peer) + // is absent, not silent: it must not be read as this peer having answered. + const phaseResponded = (phase: CatchupPhaseProgress | null | undefined): boolean => { + if (!phase) return false; const progress = classifyDurableProgress(phase); if (progress.transportFailed) return false; if (!progress.deferredByBackpressure) return true; @@ -546,7 +904,7 @@ export function catchupPeerResponded( || (phase.insertedMetaTriples ?? 0) > 0 || (phase.insertedDataTriples ?? phase.insertedTriples ?? 0) > 0; }; - return phaseResponded(durable) || Boolean(shared && phaseResponded(shared)); + return phaseResponded(durable) || phaseResponded(shared); } export interface CatchupRunner { @@ -596,6 +954,8 @@ class WorkerCatchupRunner implements CatchupRunner { private readonly worker: Worker; private nextRunId = 0; private readonly pendingRuns = new Map(); + /** Set once the worker dies; every later run fails fast instead of hanging. */ + private workerFailure: Error | undefined; constructor(private readonly agent: DKGAgent) { const jsWorkerUrl = new URL('./catchup-runner-worker-impl.js', import.meta.url); @@ -616,12 +976,31 @@ class WorkerCatchupRunner implements CatchupRunner { } }); this.worker.on('error', (error) => { - for (const [, pending] of this.pendingRuns) pending.reject(error); - this.pendingRuns.clear(); + this.fail(error); }); + // `close()` terminates the worker, which emits 'exit' — never 'error'. A + // crashed or terminated worker is also permanent: the runner is constructed + // once per daemon and `postMessage` to a dead worker neither throws nor + // delivers. Without this, an in-flight run stayed pending forever AND every + // later run did too, so the daemon's fire-and-forget subscribe jobs were + // pinned at `running` with no `finishedAt` for the rest of the process's + // life — and the route's dedupe then hands that stuck job back on every + // re-subscribe, so an operator cannot even retrigger. + this.worker.on('exit', (code) => { + this.fail(new Error(`Catch-up worker exited (code ${code}) before the run completed`)); + }); + } + + /** Latch the terminal failure and settle everything waiting on the worker. */ + private fail(error: Error): void { + this.workerFailure ??= error; + const pending = [...this.pendingRuns.values()]; + this.pendingRuns.clear(); + for (const run of pending) run.reject(error); } run(request: CatchupRunRequest): Promise { + if (this.workerFailure) return Promise.reject(this.workerFailure); const runId = this.nextRunId++; return new Promise((resolve, reject) => { this.pendingRuns.set(runId, { resolve, reject }); @@ -652,7 +1031,28 @@ class WorkerCatchupRunner implements CatchupRunner { case 'prepareCatchup': { const [contextGraphId] = args as [string]; const isPrivateContextGraph = await agent.isPrivateContextGraph(contextGraphId); - const preferredPeerId = await agent.resolvePreferredSyncPeerId(contextGraphId); + // ONE resolution, two notions. Ranking uses whatever peer is available; + // letting one peer's answer stand for the whole graph requires the + // stricter notion, because a join-approval bootstrap hint is + // authenticated but can be stale — it orders the walk without being + // allowed to end it. Resolving twice would read `_meta` twice (and run + // the registry fallback twice for a wallet-address curator) per + // catch-up, and the resolver evicts the bootstrap hint once metadata + // confirms a curator, so the second call is not the same call. + const resolution: SyncPeerResolution = + typeof agent.resolveSyncPeerWithProvenance === 'function' + ? await agent.resolveSyncPeerWithProvenance(contextGraphId) + : { + peerId: await agent.resolvePreferredSyncPeerId(contextGraphId), + // An agent without the provenance resolver cannot establish + // authority, and must not be assumed to have it. + provenance: 'bootstrap-hint', + }; + const preferredPeerId = resolution.peerId; + // The agent's own definition of "may end the walk", not a restatement + // of it: a renamed or added provenance value has to break here rather + // than silently downgrade every curator to non-authoritative. + const authoritativePeerId = authoritativeSyncPeerId(resolution); if (preferredPeerId) { await agent.ensurePeerConnected(preferredPeerId); } @@ -668,6 +1068,7 @@ class WorkerCatchupRunner implements CatchupRunner { return { preferredPeerId, + authoritativePeerId, isPrivateContextGraph, peerIds, connectedPeers: peerIds.length, @@ -681,22 +1082,40 @@ class WorkerCatchupRunner implements CatchupRunner { return agent.waitForSyncProtocol({ toString: () => peerId }); } case 'syncDurable': { - const [peerId, contextGraphId, priority] = args as [string, string, number | undefined]; + const [peerId, contextGraphId, priority, source] = args as [ + string, string, number | undefined, unknown, + ]; return agent.syncFromPeerDetailed( peerId, [contextGraphId], undefined, undefined, undefined, - priority === undefined ? undefined : { priority }, + { + ...(priority === undefined ? {} : { priority }), + // This RPC argument crossed a structured-clone boundary, so its + // compile-time type guaranteed nothing. Clamp it to the closed + // diagnostic set HERE, at the untrusted edge, so every in-process + // caller past it is typed `SyncAdmissionSource`. + source: normalizeSyncAdmissionSource( + typeof source === 'string' ? source : undefined, + ), + }, ); } case 'syncSharedMemory': { - const [peerId, contextGraphId, priority] = args as [string, string, number | undefined]; + const [peerId, contextGraphId, priority, source] = args as [ + string, string, number | undefined, unknown, + ]; return agent.syncSharedMemoryFromPeerDetailed( peerId, [contextGraphId], - priority === undefined ? undefined : { priority }, + { + ...(priority === undefined ? {} : { priority }), + source: normalizeSyncAdmissionSource( + typeof source === 'string' ? source : undefined, + ), + }, ); } case 'finalizeCatchup': { diff --git a/packages/cli/src/cli-helpers.ts b/packages/cli/src/cli-helpers.ts index 30473b29e3..d572fd5544 100644 --- a/packages/cli/src/cli-helpers.ts +++ b/packages/cli/src/cli-helpers.ts @@ -208,8 +208,10 @@ function printCatchupStatus(status: Awaited 0 ? `, ${notAttempted} not needed` : ''}), data ${status.result.dataSynced}, shared memory ${status.result.sharedMemorySynced}`, ); if (status.result.deferredBackpressure > 0) { console.log(`Deferred: ${status.result.deferredBackpressure} phase(s) by local scheduler backpressure`); diff --git a/packages/cli/src/context-graph-readiness.ts b/packages/cli/src/context-graph-readiness.ts index 18c9e39a4b..91aab03a66 100644 --- a/packages/cli/src/context-graph-readiness.ts +++ b/packages/cli/src/context-graph-readiness.ts @@ -6,7 +6,12 @@ import type { } from '@origintrail-official/dkg-node-ui'; import { catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByAuthorityHostedEmpty, + catchupPlaneProvenByData, + catchupPlaneProvenByUnanimousEmpty, + catchupPlaneReady, type CatchupJobResult, + type CatchupPlaneCompletionEvidence, } from './catchup-runner.js'; export { catchupPlaneCompletedWithoutFailure } from './catchup-runner.js'; @@ -136,16 +141,23 @@ function catchupServedUsableData(result: CatchupJobResult): boolean { return result.dataSynced > 0 || result.sharedMemorySynced > 0; } +/** + * Did ANY peer complete this plane cleanly, whatever it carried? + * + * Every carrier of clean-completion evidence must be listed here, not just the + * ones that prove readiness: this predicate gates the denial and no-response + * branches that run BEFORE `catchupPlaneReady` is ever consulted, so a form of + * evidence missing from it is silently unreachable. The curator's hosted-empty + * round is the newest carrier and is exactly that shape — no data, no wire-empty + * response, and still a clean answer from the one peer that speaks for the graph. + */ function cleanCompletionHasResponse( - completion: { - verifiedDataPeers: number; - verifiedPrivateOnlyPeers?: number; - emptyPeers: number; - } | undefined, + completion: CatchupPlaneCompletionEvidence | undefined, ): boolean { return (completion?.verifiedDataPeers ?? 0) > 0 || (completion?.verifiedPrivateOnlyPeers ?? 0) > 0 || - (completion?.emptyPeers ?? 0) > 0; + (completion?.emptyPeers ?? 0) > 0 || + (completion?.authorityEmptyPeers ?? 0) > 0; } function catchupHasRequestedCleanPeerResponse( @@ -175,31 +187,86 @@ export function catchupResultHasCleanResponse(result: CatchupJobResult): boolean (!result.denied && peerReturnedMetadata); } -function catchupPlaneReadyThisRun(input: { +interface CatchupPlaneReadinessThisRun { + /** Whether this plane counts as ready for THIS run's reported job status. */ + ready: boolean; + /** + * Whether the evidence is strong enough to PERSIST as sticky readiness + * provenance. + * + * Readiness provenance is carried forward by an OR against + * `readinessBeforeCatchup`, so anything recorded here is permanent for the + * subscription. Verified content earns it outright, as does the curator's own + * word that it hosts an empty graph. + * + * A unanimous-empty round earns it only when the round was FULLY ACCOUNTED: + * every peer the walk attempted actually answered (`failedPeers === 0`). + * Emptiness is a verdict derived from ABSENCE of evidence, so it is only as + * good as the denominator it was taken over — with peers unaccounted for and + * no authoritative curator to anchor it, a single unrelated empty response + * produces the same verdict as a genuinely empty graph. + * + * Splitting it this way keeps both properties that pulled against each other: + * + * - LIVENESS. The per-run verdict is unchanged, so a graph on a lossy network + * still reports `done` instead of retrying forever. Failing the verdict + * itself closed on unaccounted peers was rejected for exactly that reason. + * - NO FROZEN GUESS. Nothing derived from a partial round is written down, so + * a wrong empty verdict cannot outlive the run that produced it. + * + * This bit is what `statePatch.synced` is built from, and `synced` gates + * write preflight (`contextGraphRowIsWritable`), so anything admitted here + * grants durable readiness to consumers that never see the job result. + */ + persistable: boolean; +} + +function catchupPlaneReadinessThisRun(input: { result: CatchupJobResult; plane: 'durable' | 'sharedMemory'; isPrivate: boolean; -}): boolean { +}): CatchupPlaneReadinessThisRun { + const diagnostics = input.result.diagnostics?.[input.plane]; const completion = input.result.cleanPlaneCompletions?.[input.plane]; + const options = { isPrivate: input.isPrivate }; + // Every attempted peer answered, so the empty verdict was taken over the + // whole peer set rather than over whoever happened to reply. + const fullyAccounted = (diagnostics?.failedPeers ?? 0) === 0; if (completion) { - const verifiedPrivateOnly = input.plane === 'durable' && - (input.result.cleanPlaneCompletions?.durable.verifiedPrivateOnlyPeers ?? 0) > 0; - return completion.verifiedDataPeers > 0 || - verifiedPrivateOnly || - (!input.isPrivate && completion.emptyPeers > 0); + const provenPositively = catchupPlaneProvenByData(completion) + || catchupPlaneProvenByAuthorityHostedEmpty(completion, diagnostics, options); + const unanimousEmpty = catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options); + return { + ready: provenPositively || unanimousEmpty, + persistable: provenPositively || (unanimousEmpty && fullyAccounted), + }; } // Backward compatibility for callers that construct a legacy result (for // example, an older in-process runner during a rolling upgrade). New worker // results always carry cleanPlaneCompletions, so aggregate failures are not - // used as readiness evidence on the production path. - const diagnostics = input.result.diagnostics?.[input.plane]; + // used as readiness evidence on the production path. The same fail-closed + // rule applies: aggregate counters can show that SOMEBODY answered empty, but + // only a content-free, failure-free round proves the plane really is empty. const dataProgress = input.plane === 'durable' ? input.result.dataSynced > 0 || (input.result.diagnostics?.durable.verifiedPrivateOnlyResponses ?? 0) > 0 : input.result.sharedMemorySynced > 0; - return catchupPlaneCompletedWithoutFailure(diagnostics) && - (dataProgress || (!input.isPrivate && (diagnostics?.emptyResponses ?? 0) > 0)); + if (catchupPlaneCompletedWithoutFailure(diagnostics) && dataProgress) { + return { ready: true, persistable: true }; + } + // Pass NO completion evidence rather than an all-zero stand-in: the empty + // proof consults the raw aggregate counters only when completion evidence is + // genuinely absent, and a synthetic `emptyPeers: 0` would read as "the + // per-peer view saw no clean empty response" and suppress the legacy path. + const ready = catchupPlaneReady(undefined, diagnostics, options); + return { + ready, + // No completion evidence means neither positive proof mode can fire, so + // anything true here came from the aggregate empty counter and is subject + // to the same fully-accounted requirement. + persistable: ready && fullyAccounted, + }; } export interface ContextGraphCatchupReadinessClassification { @@ -256,25 +323,43 @@ export function classifyContextGraphCatchupReadiness(input: { }; } - const durableReadyThisRun = catchupPlaneReadyThisRun({ + const durableThisRun = catchupPlaneReadinessThisRun({ result, plane: 'durable', isPrivate: input.isPrivate, }); - const sharedMemoryReadyThisRun = input.includeSharedMemory && - catchupPlaneReadyThisRun({ + const sharedMemoryThisRun = input.includeSharedMemory + ? catchupPlaneReadinessThisRun({ result, plane: 'sharedMemory', isPrivate: input.isPrivate, - }); + }) + : { ready: false, persistable: false }; + const durableReadyThisRun = durableThisRun.ready; + const sharedMemoryReadyThisRun = sharedMemoryThisRun.ready; const currentReadinessProvenance = input.readinessBeforeCatchup.version >= CONTEXT_GRAPH_READINESS_VERSION; - const durableVerified = - (currentReadinessProvenance && input.readinessBeforeCatchup.durableVerified) || - durableReadyThisRun; - const sharedMemoryVerified = - (currentReadinessProvenance && input.readinessBeforeCatchup.sharedMemoryVerified) || - sharedMemoryReadyThisRun; + const durableVerifiedBefore = + currentReadinessProvenance && input.readinessBeforeCatchup.durableVerified; + const sharedMemoryVerifiedBefore = + currentReadinessProvenance && input.readinessBeforeCatchup.sharedMemoryVerified; + const durableVerified = durableVerifiedBefore || durableReadyThisRun; + const sharedMemoryVerified = sharedMemoryVerifiedBefore || sharedMemoryReadyThisRun; + // What this run is allowed to FREEZE, as opposed to what it reports. These + // diverge only for a unanimous-empty verdict, which stays re-derived per run + // so that a wrong empty verdict cannot become permanent. + const durableVerifiedPersisted = durableVerifiedBefore || durableThisRun.persistable; + const sharedMemoryVerifiedPersisted = + sharedMemoryVerifiedBefore || sharedMemoryThisRun.persistable; + // `subscription.synced` is a SECOND persisted readiness bit, living outside + // the provenance store and consumed by callers that never see this job's + // result — `contextGraphRowIsWritable` treats `subscribed && synced` as + // writable. It must therefore carry the same verdict as `readinessPatch`, + // not the transient one. The pre-catch-up path already assumes they agree: + // it derives `synced` from the persisted provenance and patches the row + // back into line, so letting them diverge here would be corrected away on + // the next pass anyway — after a window in which the graph looked writable. + const overallVerifiedPersisted = durableVerifiedPersisted || sharedMemoryVerifiedPersisted; const overallVerified = durableVerified || sharedMemoryVerified; const missingGraphProof = !overallVerified; const missingRequestedSharedMemory = @@ -302,14 +387,14 @@ export function classifyContextGraphCatchupReadiness(input: { jobStatus, error, statePatch: { - synced: overallVerified, - sharedMemorySynced: sharedMemoryVerified, + synced: overallVerifiedPersisted, + sharedMemorySynced: sharedMemoryVerifiedPersisted, metaSynced: true, pendingMeta: false, }, readinessPatch: { - durableVerified, - sharedMemoryVerified, + durableVerified: durableVerifiedPersisted, + sharedMemoryVerified: sharedMemoryVerifiedPersisted, }, eventPayload: durableReadyThisRun || sharedMemoryReadyThisRun ? { diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index c8688098fa..d0d401fec4 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -1,21 +1,53 @@ // catchup-runner-worker-impl.test.ts // -// Drives the daemon-side Worker catch-up implementation — the -// `/api/context-graph/subscribe` path that fans a full durable+SWM sync out -// over every sync-capable peer — over a mocked `parentPort`, and pins the -// 2026-07-07 sync-storm mitigation (C-1) at THIS call site: no more than -// CATCHUP_MAX_CONCURRENT_PEER_SYNCS per-peer sync rounds (or protocol probes) -// may ever be in flight, while every peer still gets synced exactly once, the -// aggregation keeps its one-result-per-peer input-order shape, and one peer's -// failure stays isolated instead of failing the whole run. -import { describe, expect, it, vi } from 'vitest'; +// Drives the daemon-side Worker catch-up implementation — the production +// `/api/context-graph/subscribe` path — over a mocked `parentPort`, and pins +// two guarantees at THIS call site: +// +// * the 2026-07-07 sync-storm mitigation (C-1): no more than +// CATCHUP_MAX_CONCURRENT_PEER_SYNCS per-peer sync rounds (or protocol +// probes) may ever be in flight, per-peer failures stay isolated, and the +// aggregation keeps its one-result-per-peer input-order shape; +// * the issue #2006 progressive walk: peers are contacted in escalating +// waves over the authority-ranked list and the walk STOPS as soon as the +// RESOLVED CURATOR has settled every requested plane, so the happy path +// transfers one payload instead of one per peer. A non-authoritative peer's +// clean round settles nothing — it can neither stop the walk nor narrow a +// later peer to one plane. +import { afterAll, describe, expect, it, vi } from 'vitest'; import { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, CATCHUP_MAX_CONCURRENT_PEER_SYNCS, FOREGROUND_CATCHUP_SYNC_PRIORITY, } from '@origintrail-official/dkg-agent'; import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; +// The foreground backpressure budget is wall-clock (default 180s). Shrink it +// for this file so the persistently-deferred case settles quickly; the exact +// deadline arithmetic is pinned deterministically in +// packages/agent/test/catchup-policy.test.ts with an injected clock. +// `vi.hoisted` runs before imports so the module-load-time constant picks this +// up — but it mutates the REAL process env, and vitest can reuse a worker +// process across files in a shard. Anything loaded afterwards, including a +// daemon spawned by a sibling suite, would otherwise inherit the shortened backpressure budget. +// +// The budget must stay comfortably ABOVE `CATCHUP_BACKPRESSURE_BASE_DELAY_MS` +// (250 ms). At exactly 250 ms the first backoff is clamped to the whole +// remaining budget, so the sleep ends ON the deadline and every retry in this +// file depended on the loop admitting an attempt there — which the post-sleep +// deadline check now declines. That ratio does not occur in production (the +// default budget is 180 s against the same 250 ms base), so pinning it would +// have pinned an artefact of the fixture rather than a behaviour. +const previousCATCHUPBACKPRESSUREMAXWAITMS = vi.hoisted(() => { + const before = process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = '900'; + return before; +}); + +afterAll(() => { + if (previousCATCHUPBACKPRESSUREMAXWAITMS === undefined) delete process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + else process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = previousCATCHUPBACKPRESSUREMAXWAITMS; +}); + // The worker impl wires itself to `parentPort` at module load, so a // controllable port has to be in place BEFORE the module is imported. // Everything else from node:worker_threads stays real. @@ -120,8 +152,97 @@ async function runWorkerCatchup(request: CatchupRunRequest, handler: InvokeHandl }); } +/** + * SCOPE OF THESE TESTS — read before trusting a green run. + * + * Every case below hands the worker an `authoritativePeerId` through the stubbed + * `prepareCatchup` boundary. **No production resolver route currently produces + * one.** `resolveCuratorSyncPeer` was changed in `e7f46dca2` so that nothing + * earns `metadata` provenance, because a curator-to-peer binding read out of + * accumulated `/_meta` identifies the graph that HOLDS the rows, not the + * writer that SUPPLIED them — and ordinary durable-meta catch-up lets a + * contacted peer write those very rows. + * + * So these tests verify that the worker HANDLES an authority correctly IF it is + * given one. They do NOT verify that the early stop or the per-plane narrowing + * happens in the shipped build — it cannot, and byte volume is at the pre-fix + * level until #2018 lands a trusted binding. Read as end-to-end evidence for the + * fan-out reduction they would be claiming something untrue. + * + * They are kept rather than deleted because #2018 re-enables exactly this + * machinery, and deleting them would remove the contract it has to satisfy. When + * that lands, the missing piece is a case that derives `authoritativePeerId` + * through the REAL resolver/projection path instead of injecting it here. + */ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1)', () => { - it('caps in-flight peer syncs and protocol probes at the shared limit while still syncing every peer in input order', async () => { + it('stops the walk at the first peer that proves every requested plane', async () => { + const peerIds = Array.from({ length: 20 }, (_, i) => `peer-${i}`); + const durableOrder: string[] = []; + const sharedSeen: string[] = []; + const probeOrder: string[] = []; + const syncPriorities: Array = []; + const syncSources: Array = []; + const finalizeCalls: unknown[][] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-one-payload', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + probeOrder.push(args[0] as string); + return true; + case 'syncDurable': + durableOrder.push(args[0] as string); + syncPriorities.push(args[2] as number | undefined); + syncSources.push(args[3] as string | undefined); + return durableResult(); + case 'syncSharedMemory': + sharedSeen.push(args[0] as string); + syncPriorities.push(args[2] as number | undefined); + syncSources.push(args[3] as string | undefined); + return sharedResult(); + case 'finalizeCatchup': + finalizeCalls.push(args); + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The whole point of issue #2006: one authoritative payload, not twenty. + // Before the progressive walk this was `toEqual(peerIds)` on both planes — + // 20 full durable pulls and 20 full SWM pulls for a single graph. + expect(durableOrder).toEqual(['peer-0']); + expect(sharedSeen).toEqual(['peer-0']); + expect(syncPriorities).toEqual([ + FOREGROUND_CATCHUP_SYNC_PRIORITY, + FOREGROUND_CATCHUP_SYNC_PRIORITY, + ]); + expect(syncSources).toEqual(['catchup-foreground', 'catchup-foreground']); + + // Probing stays eager over the whole connected set: `syncCapablePeers` and + // `noProtocolPeers` are read by daemon status mapping as counts over every + // connected peer, not over the walked prefix. + expect(probeOrder.sort()).toEqual([...peerIds].sort()); + expect(result.syncCapablePeers).toBe(peerIds.length); + expect(result.selectedPeers).toBe(peerIds.length); + + // Peers the walk deliberately skipped are neither tried nor failed. + expect(result.peersTried).toBe(1); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.peersResponded).toBe(1); + expect(result.peersSucceeded).toBe(1); + expect(result.diagnostics?.durable.failedPeers).toBe(0); + expect(result.diagnostics?.durable.timedOutPhases).toBe(0); + + expect(result.deferredBackpressure).toBe(0); + expect(result.dataSynced).toBe(1); + expect(result.sharedMemorySynced).toBe(1); + expect(result.denied).toBe(false); + expect(finalizeCalls).toEqual([['cg-one-payload', 1, 1]]); + }); + + it('escalates waves and still caps in-flight peer syncs when no peer proves the plane', async () => { const peerIds = Array.from({ length: 20 }, (_, i) => `peer-${i}`); // The bound is only observable when the peer set exceeds the cap. expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeLessThan(peerIds.length); @@ -131,11 +252,13 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) let inFlightSyncs = 0; let peakSyncs = 0; const durableOrder: string[] = []; - const sharedSeen: string[] = []; - const syncPriorities: Array = []; - const finalizeCalls: unknown[][] = []; + const startOrder: string[] = []; + + // Nobody completes, so nothing is ever proven and the walk must cover the + // whole peer set — the fallback path. + const unprovenDurable = () => ({ ...durableResult(), complete: false }); - const result = await runWorkerCatchup({ contextGraphId: 'cg-storm', includeSharedMemory: true }, async (method, args) => { + const result = await runWorkerCatchup({ contextGraphId: 'cg-storm', includeSharedMemory: false }, async (method, args) => { switch (method) { case 'prepareCatchup': return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; @@ -148,56 +271,943 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) } case 'syncDurable': { durableOrder.push(args[0] as string); - syncPriorities.push(args[2] as number | undefined); + startOrder.push(args[0] as string); inFlightSyncs += 1; peakSyncs = Math.max(peakSyncs, inFlightSyncs); await delay(4); inFlightSyncs -= 1; - return durableResult(); - } - case 'syncSharedMemory': { - sharedSeen.push(args[0] as string); - syncPriorities.push(args[2] as number | undefined); - inFlightSyncs += 1; - peakSyncs = Math.max(peakSyncs, inFlightSyncs); - await delay(2); - inFlightSyncs -= 1; - return sharedResult(); + return unprovenDurable(); } case 'finalizeCatchup': - finalizeCalls.push(args); return null; default: throw new Error(`unexpected invoke: ${method}`); } }); - // The storm guard: neither fan-out phase ever exceeds the cap… + // The storm guard: neither phase ever exceeds the cap… expect(peakSyncs).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); expect(peakProbes).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); - // …but the fan-out is still actually parallel, not accidentally serialised. + // …but later waves are still actually parallel, not accidentally serialised. expect(peakSyncs).toBeGreaterThan(1); + // No curator resolved here, so the opening wave is NOT narrowed to one peer + // (that narrowing only buys anything when there is an authority to spend it + // on). The ranked order is still honoured. + expect(startOrder[0]).toBe('peer-0'); - // Coverage preserved: every peer synced exactly once, started in input - // order (the bounded mapper's shared cursor hands out work in order). + // Coverage preserved when nothing proves: every peer walked, in rank order. expect(durableOrder).toEqual(peerIds); - expect([...sharedSeen].sort()).toEqual([...peerIds].sort()); - expect(syncPriorities).toEqual( - Array.from({ length: peerIds.length * 2 }, () => FOREGROUND_CATCHUP_SYNC_PRIORITY), - ); - - // Aggregation unchanged from the unbounded Promise.all shape. - expect(result.selectedPeers).toBe(peerIds.length); - expect(result.syncCapablePeers).toBe(peerIds.length); expect(result.peersTried).toBe(peerIds.length); - expect(result.peersResponded).toBe(peerIds.length); - expect(result.peersSucceeded).toBe(peerIds.length); - expect(result.deferredBackpressure).toBe(0); + expect(result.peersNotAttempted).toBe(0); expect(result.dataSynced).toBe(peerIds.length); - expect(result.sharedMemorySynced).toBe(peerIds.length); - expect(result.denied).toBe(false); - expect(result.diagnostics?.durable.failedPeers).toBe(0); - expect(finalizeCalls).toEqual([['cg-storm', peerIds.length, peerIds.length]]); + }); + + it('keeps walking past a non-authoritative peer that returned verified data', async () => { + // A peer's `complete` flag proves it served ITS OWN manifest, not the union + // of what the network holds: peer-0 can cleanly return KA-1 while a later + // peer holds KA-2 for the same graph. Without a resolved curator there is no + // reference snapshot, so a clean data-bearing round must NOT cut the walk + // short and strand the other peers' Knowledge Assets. + // + // The peer set must exceed the concurrency cap, or the whole walk is one + // wave and this assertion could not fail regardless of the gate. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-disjoint', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.dataSynced).toBe(peerIds.length); + }); + + it('does not let a non-authoritative peer narrow a later peer to one plane', async () => { + // The other half of the authority gate: a non-curator peer settling shared + // memory must not cause later peers to be contacted for durable only. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + await runWorkerCatchup({ contextGraphId: 'cg-no-narrow', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { ...durableResult(), complete: false }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + }); + + it('stops after the curator proves the only requested plane', async () => { + // Plain `subscribe` with no workspace is the most common production shape + // for the early stop; the multi-wave curator case above requests both + // planes, so this pins the durable-only path. + const peerIds = Array.from({ length: 10 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-durable-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + }); + + it('lets the curator settle the DURABLE plane by answering cleanly empty', async () => { + // The Context Graph is the curator's, so its "there is nothing here" is + // authoritative for the durable plane and one payload settles it. The + // shared-memory plane is deliberately NOT symmetric — see the next test. + const peerIds = Array.from({ length: 10 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-durable', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(1); + }); + + it('does NOT let the curator settle the shared-memory plane by answering empty', async () => { + // Shared memory is a per-agent-address layered union + // (`//`) contributed by many members, so a curator that + // holds no SWM rows has not said anything about the members' layers — it + // does not own them. Settling on its silence skipped peers that held valid + // rows, and could report `sharedMemoryVerified` with `sharedMemorySynced: 0`. + // + // The durable plane still settles on the curator's round, so the expensive + // half of the walk is still one payload: fallback peers are narrowed to SWM. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-union', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + // The curator has nothing; a later member holds the rows. + if (args[0] === 'peer-0') { + return { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The expensive plane is still pulled once… + expect(durableCalls).toEqual(['peer-0']); + // …while the union plane keeps walking, and reaches the member that has rows. + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); + expect(result.cleanPlaneCompletions?.sharedMemory.verifiedDataPeers).toBeGreaterThan(0); + expect(result.sharedMemorySynced).toBeGreaterThan(0); + }); + + it('skips the durable plane on fallback peers once the curator settled it', async () => { + // The `durable: null` round — the reason the peer-accounting helpers accept + // a missing plane at all. The curator settles durable but not shared + // memory, so later peers must be contacted for shared memory only and must + // not be credited with a durable response. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + const sharedPriorities: Array = []; + const sharedSources: Array = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-fallback', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + sharedPriorities.push(args[2] as number | undefined); + sharedSources.push(args[3] as string | undefined); + // The curator engages and fails (so SWM is never settled); every + // fallback peer transport-fails, delivering nothing at all. + return args[0] === 'peer-0' + ? { + ...sharedResult(), + insertedTriples: 0, + insertedDataTriples: 0, + completedPhases: 0, + failedPhases: 1, + } + : { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 0, + failedPeers: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Durable pulled once, from the curator; shared memory from everyone. + expect(durableCalls).toEqual(['peer-0']); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + // The shared-only fallback goes through a different call path than the + // both-planes one, so it has to carry foreground admission itself. + expect(sharedPriorities).toEqual( + peerIds.map(() => FOREGROUND_CATCHUP_SYNC_PRIORITY), + ); + expect(sharedSources).toEqual(peerIds.map(() => 'catchup-foreground')); + expect(result.peersTried).toBe(peerIds.length); + expect(result.peersNotAttempted).toBe(0); + // The skipped durable plane must not manufacture a response for peers whose + // only requested plane transport-failed: only the curator responded. + expect(result.peersResponded).toBe(1); + expect(result.peersSucceeded).toBe(0); + // One durable round in the whole walk — that is the amplification fix. + expect(result.diagnostics?.durable.fetchedDataTriples).toBe(1); + expect(result.dataSynced).toBe(1); + }); + + it('does not let an empty curator round settle a PRIVATE plane', async () => { + // Readiness deliberately refuses to prove a private plane from an empty + // response, so stopping the walk on one would strand it: fallback peers + // that may hold authorized private data are skipped and a recoverable + // catch-up turns into `unreachable`. Emptiness only settles public planes. + const peerIds = ['peer-curator', 'peer-with-data', 'peer-c', 'peer-d', 'peer-e', 'peer-f']; + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-empty', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + if (args[0] === 'peer-curator') { + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The authorized fallback peer must still be reached. + expect(durableCalls).toContain('peer-with-data'); + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBeGreaterThan(0); + }); + + it('still lets a verified private-only curator round settle a private plane', async () => { + // The complement: a cryptographically verified V2 response whose public + // graph is intentionally empty is CONTENT, not emptiness, and must keep + // working as positive proof on a private graph. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 8, + fetchedMetaTriples: 8, + fetchedDataTriples: 0, + insertedMetaTriples: 8, + insertedDataTriples: 0, + verifiedPrivateOnlyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.durable.verifiedPrivateOnlyPeers).toBe(1); + }); + + it('does not let an empty curator round settle a PRIVATE shared-memory plane', async () => { + // The shared-memory half of the private rule. It needs its own coverage: + // `includeSharedMemory` defaults to true on subscribe and shared memory is + // frequently empty, so this is the plane an over-eager empty rule would + // settle first — stranding the walk before any authorized peer holding SWM + // data is contacted. The durable plane settles by verified content here, so + // only the shared plane's rule is under test. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-swm', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 8, + fetchedMetaTriples: 8, + fetchedDataTriples: 0, + insertedMetaTriples: 8, + insertedDataTriples: 0, + verifiedPrivateOnlyResponses: 1, + }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Durable settled on the curator's verified content and is not re-pulled… + expect(durableCalls).toEqual(['peer-0']); + // …while the unproven shared plane keeps walking every remaining peer. + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + // The shared plane produces no authority evidence at all now — privacy is no + // longer the only thing standing between an empty curator round and a + // settled SWM plane. + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); + }); + + it('reports the curator as unanswered when it was selected and transport-failed', async () => { + // The exact #2006 shape, produced by the walk's own design: a resolvable + // curator is ranked first and gets wave 1 ALONE, so when it transport-fails + // the walk moves on to strangers, one answers empty, and without this signal + // that stranger's silence would settle a 40-KA graph as `done` with zero. + const peerIds = ['peer-curator', 'peer-a', 'peer-b', 'peer-c']; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-curator-silent', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + if (args[0] === 'peer-curator') { + // Transport failure: no clean completion from the one peer that knows. + return { ...durableResult(), complete: false, insertedTriples: 0, + fetchedDataTriples: 0, insertedDataTriples: 0, bytesReceived: 0, + completedPhases: 0, failedPeers: 1 }; + } + return { ...durableResult(), insertedTriples: 0, fetchedDataTriples: 0, + insertedDataTriples: 0, bytesReceived: 0, completedPhases: 2, emptyResponses: 1 }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(result.diagnostics?.durable.authorityUnanswered).toBe(true); + // Strangers still answered cleanly empty — that is exactly what must NOT + // settle the plane now. + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBeGreaterThan(0); + }); + + it('reports the curator as answered when it completed cleanly', async () => { + // The complement, so the flag cannot be hardwired true: a curator that + // answers must leave the round provable. + const peerIds = ['peer-curator', 'peer-a']; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-curator-answered', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(result.diagnostics?.durable.authorityUnanswered).toBe(false); + }); + + it('does not stop on a hosted-empty curator that the round already contradicted', async () => { + // The curator says "nothing here" while another peer in the SAME wave served + // content that failed verification. Readiness treats that as content + // EXISTING, so it voids the empty proof — and if the walk had already + // stopped on the curator's word, the job ends unready having skipped peers + // that might have delivered valid data. Worst of both. + // + // The curator is deliberately NOT first, so the opening wave is full width + // and both responses land in the same wave: this is exactly the ordering + // where a per-peer stop decision cannot see the contradiction. + // `peer-later` MUST sit in a later wave: with the curator not first the + // opening wave is full width, so a three-peer list would contact everyone + // regardless and the test could not observe an early stop at all. + const wave1 = ['peer-rejected', 'peer-curator', 'peer-quiet-a', 'peer-quiet-b']; + const peerIds = [...wave1, 'peer-later', 'peer-quiet-c']; + expect(wave1.length).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-contradicted', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + if (args[0] === 'peer-curator') { + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + } + if (args[0] === 'peer-rejected') { + // Served content for this graph; verification threw it out. + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 4_000, + insertedDataTriples: 0, + rejectedKcs: 1, + }; + } + if (args[0] === 'peer-later') return durableResult(); + // Everyone else answers content-free, so the only verified data in the + // run is the one behind the wave boundary. + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 1, + completedPhases: 2, + }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The contradiction is visible round-wide, so the curator's emptiness does + // not settle the plane and the remaining peer is still reached. + expect(durableCalls).toContain('peer-later'); + expect(result.peersNotAttempted).toBe(0); + // …and that last peer's verified data is what actually proves the plane. + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBeGreaterThan(0); + }); + + it('does not settle the SHARED-MEMORY plane on curator metadata alone', async () => { + // End-to-end counterpart of the plane-aware reducer: shared memory is + // contributed by many members rather than owned by the curator, so + // `insertedMetaTriples` there is not the hosting proof `/_meta` is on + // the durable plane. Settling on it would stop the walk before any member + // holding the SWM rows is contacted. Public graph, so privacy is not what + // is doing the work here. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-meta', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return { + ...sharedResult(), + insertedTriples: 5, + insertedMetaTriples: 5, + insertedDataTriples: 0, + fetchedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 0, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); + }); + + it('settles a public plane when the CURATOR hosts the graph and has no data', async () => { + // A registered public Context Graph with no Knowledge Assets yet. Its host + // still serves the CG definition triples from `/_meta`, so it answers + // metadata-only — never wire-empty — and no whole-round emptiness rule can + // fire for it. The curator saying "I host this and there is nothing in it" + // is the only evidence that exists, and without it such a graph would sit + // at `unreachable` forever while re-walking every peer on every retry. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-registered-empty', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(1); + // The same round from a peer that is NOT the curator proves nothing: it is + // what any member holding `_meta` but no data looks like. + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(0); + }); + + it('does not let a non-curator metadata-only round stop the walk', async () => { + // The counterpart of the test above, and the reason it is scoped to the + // curator: mid-sync members answering metadata-only are the commonest + // state on the network. Accepting theirs would resettle #2006 exactly — + // `done` with zero Knowledge Assets out of forty. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-members-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(0); + }); + + it('does not let a bootstrap-hint preferred peer stop the walk', async () => { + // `resolvePreferredSyncPeerId` falls back to the authenticated join-approval + // hint when metadata resolves no curator. That hint can be stale — a curator + // that has since rotated its libp2p identity leaves an ordinary member on + // that peer id — so it orders the walk but must never end it. The worker + // sees that as `preferredPeerId` WITHOUT `authoritativePeerId`. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-hint-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + }); + + it('opens at the full concurrency cap when no curator resolved', async () => { + // A single-peer opening wave buys "one payload from the curator". Without a + // resolvable curator it buys nothing, so the walk must not serialise the + // head of the list and pay an extra round-trip on every round. + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + let inFlight = 0; + let peak = 0; + const startOrder: string[] = []; + + await runWorkerCatchup({ contextGraphId: 'cg-no-curator', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + startOrder.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(4); + inFlight -= 1; + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(startOrder).toEqual(peerIds); + }); + + it('walks a no-authority round as ONE pass, with no barrier between waves', async () => { + // Waves exist only so an authority can cut the walk short. With no + // authoritative curator nothing can break the loop, so splitting the peer + // set into waves saves no fetch and only adds a barrier — making the round + // slower than the single bounded pass it replaced. + // + // Barriers are invisible to a peak-concurrency assertion (both shapes peak + // at the cap), so this pins the property that actually differs: with a + // sliding window a LATER peer starts while an early slow peer is still in + // flight; behind a barrier it cannot. + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + let slowPeerInFlight = false; + let startedDuringSlowPeer = 0; + + await runWorkerCatchup( + { contextGraphId: 'cg-no-authority-single-pass', includeSharedMemory: false }, + async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: undefined, + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + const peerId = args[0] as string; + if (peerId === 'peer-0') { + slowPeerInFlight = true; + await delay(40); + slowPeerInFlight = false; + return { ...durableResult(), complete: false }; + } + // Anything beyond the first wave-width proves the window slid. + if (slowPeerInFlight && Number(peerId.slice('peer-'.length)) >= CATCHUP_MAX_CONCURRENT_PEER_SYNCS) { + startedDuringSlowPeer += 1; + } + await delay(1); + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }, + ); + + expect(startedDuringSlowPeer).toBeGreaterThan(0); + }); + + it('spends the single-peer opening wave only on a sync-capable curator', async () => { + // A REAL authority that is offline: metadata resolved a curator, so + // `authoritativePeerId` is set, but the protocol probe filters it out. The + // opening wave narrows to one peer only when the authority is the peer that + // wave would actually contact — otherwise the walk would serialise an + // arbitrary fallback peer for nothing. + // + // The fixture must set `authoritativePeerId`: without it the walk takes the + // no-curator branch (covered separately above) and neither half of the + // guard is exercised. + const peerIds = ['peer-curator', 'peer-a', 'peer-b', 'peer-c', 'peer-d']; + const durableCalls: string[] = []; + let inFlight = 0; + let peak = 0; + + await runWorkerCatchup({ contextGraphId: 'cg-curator-offline', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return args[0] !== 'peer-curator'; + case 'syncDurable': { + durableCalls.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(4); + inFlight -= 1; + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + // The offline authority is never contacted, and every reachable peer is. + expect(durableCalls).not.toContain('peer-curator'); + expect([...durableCalls].sort()).toEqual(['peer-a', 'peer-b', 'peer-c', 'peer-d']); + }); + + it('narrows fallback peers to the planes the curator already settled', async () => { + // The curator settles shared memory but never settles durable (its durable + // round engages and fails). Without per-plane narrowing, walking on for + // durable would drag a full re-pull of the ALREADY SETTLED shared-memory + // plane out of every remaining peer — the exact amplification this removes. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-only', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + complete: false, + insertedTriples: 0, + insertedDataTriples: 0, + completedPhases: 0, + timedOutPhases: 1, + }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Shared memory is settled by the curator and never pulled again… + expect(sharedCalls).toEqual(['peer-0']); + // …while the unsettled durable plane still walks everyone. + expect(durableCalls).toEqual(peerIds); + expect(result.sharedMemorySynced).toBe(1); + expect(result.cleanPlaneCompletions?.sharedMemory.verifiedDataPeers).toBe(1); }); it('keeps per-peer failure isolation and probe filtering under the bounded fan-out', async () => { @@ -218,7 +1228,10 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) durableCalls.push(args[0] as string); await delay(2); if (args[0] === 'peer-3') throw new Error('peer 3 exploded'); - return durableResult(); + // `complete: false` keeps every peer unproven, so the walk covers the + // whole set and the isolation claim below is actually exercised + // instead of being skipped by an early stop. + return { ...durableResult(), complete: false }; } case 'syncSharedMemory': sharedCalls += 1; @@ -235,6 +1248,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(sharedCalls).toBe(0); expect(result.syncCapablePeers).toBe(11); expect(result.peersTried).toBe(11); + expect(result.peersNotAttempted).toBe(0); expect(result.peersResponded).toBe(10); expect(result.peersSucceeded).toBe(10); expect(result.dataSynced).toBe(10); @@ -243,6 +1257,75 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.diagnostics?.durable.failedPeers).toBe(1); }); + it('does not let an unrelated peer\'s clean empty response stop the walk or prove readiness', async () => { + // The reported #2006 shape: a data-bearing peer fails part-way, an + // unrelated peer that has never heard of the graph answers empty. On the + // wire those two peers are indistinguishable, so the empty answer must not + // stop the walk and must not settle the job as `done`. + // + // The empty peers fill the ENTIRE first wave and the data-bearing peer sits + // behind a wave boundary, so a regression that accepted any clean-empty + // round as proof would stop before ever reaching it. A same-wave setup + // could not observe that. + const emptyPeers = Array.from( + { length: CATCHUP_MAX_CONCURRENT_PEER_SYNCS }, + (_, i) => `peer-empty-${i}`, + ); + const peerIds = [...emptyPeers, 'peer-data-failed', 'peer-quiet']; + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-mask', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + if (args[0] === 'peer-data-failed') { + return { + ...durableResult(), + complete: false, + fetchedDataTriples: 5_000, + insertedTriples: 0, + insertedDataTriples: 0, + completedPhases: 0, + timedOutPhases: 1, + failedPhases: 1, + }; + } + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Emptiness is never a stop condition, so the walk crosses the wave + // boundary and still reaches the data-bearing peer. + expect(durableCalls).toEqual(peerIds); + expect(durableCalls).toContain('peer-data-failed'); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBe(0); + // The clean-empty peers are still recorded as clean empty completions… + expect(result.cleanPlaneCompletions?.durable.emptyPeers) + .toBe(peerIds.length - 1); + // …but the round fetched data and failed, so readiness must not follow. + expect(result.diagnostics?.durable.fetchedDataTriples).toBe(5_000); + expect(result.diagnostics?.durable.failedPhases).toBe(1); + }); + it('retries only SWM after durable progress and finalizes when local pressure clears', async () => { const finalizeCalls: unknown[][] = []; let durableCalls = 0; @@ -332,7 +1415,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(callOrder).toEqual(['durable-1', 'durable-2', 'shared']); }); - it('returns deferred after a bounded durable retry budget and never starts dependent SWM', async () => { + it('returns deferred after the wall-clock durable retry budget and never starts dependent SWM', async () => { let durableCalls = 0; let sharedCalls = 0; const finalizeCalls: unknown[][] = []; @@ -368,7 +1451,12 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }, ); - expect(durableCalls).toBe(CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS.length + 1); + // The retry loop is wired and bounded: at least one retry happened (the + // pre-#2006 policy also retried, but on a fixed 850 ms ladder), and the run + // settled at the wall-clock budget instead of spinning forever. The exact + // deadline arithmetic — that attempts are governed by the clock, not by a + // fixed attempt count — is pinned in packages/agent/test/catchup-policy.test.ts. + expect(durableCalls).toBeGreaterThanOrEqual(2); expect(sharedCalls).toBe(0); expect(result.deferredBackpressure).toBe(1); expect(result.peersResponded).toBe(0); @@ -413,8 +1501,16 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.diagnostics?.durable.deniedPhases).toBe(1); expect(result.diagnostics?.sharedMemory.deniedPhases).toBe(1); expect(result.cleanPlaneCompletions).toEqual({ - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0 }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, incompleteResponders: 1 }, }); expect(result.diagnostics?.durable.verifiedPrivateOnlyResponses).toBe(0); }); @@ -425,10 +1521,14 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) async (method, args) => { switch (method) { case 'prepareCatchup': + // The denying/timing-out peer is ranked FIRST so the walk reaches + // the clean peer in a later wave: the claim under test is that a + // clean per-peer completion survives another peer's denial in the + // aggregate, which an early stop on wave 1 would never exercise. return { preferredPeerId: undefined, isPrivateContextGraph: true, - peerIds: ['peer-clean', 'peer-partial'], + peerIds: ['peer-partial', 'peer-clean'], connectedPeers: 2, }; case 'waitForSyncProtocol': @@ -459,6 +1559,10 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 1, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); @@ -497,6 +1601,10 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); @@ -542,6 +1650,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 1, emptyPeers: 0, + authorityEmptyPeers: 0, + // No `incompleteResponders`: this peer COMPLETED cleanly. The counter + // must not appear merely because a plane carried no public data. }); }); @@ -582,9 +1693,16 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); + // A clean empty round is never a stop condition — it cannot distinguish an + // empty host from a peer that never heard of the graph — so every peer is + // still walked and emptiness stays a whole-round verdict. it('records each distinct responder that cleanly completes both planes empty', async () => { const peerIds = ['peer-empty-1', 'peer-empty-2', 'peer-empty-3', 'peer-empty-4']; const result = await runWorkerCatchup( @@ -631,12 +1749,19 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result).toMatchObject({ peersResponded: peerIds.length, peersSucceeded: peerIds.length, + peersTried: peerIds.length, + peersNotAttempted: 0, dataSynced: 0, sharedMemorySynced: 0, }); expect(result.cleanPlaneCompletions).toEqual({ - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: peerIds.length }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: peerIds.length }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: peerIds.length, + authorityEmptyPeers: 0, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: peerIds.length, authorityEmptyPeers: 0 }, }); }); }); diff --git a/packages/cli/test/catchup-runner-worker-killswitch.test.ts b/packages/cli/test/catchup-runner-worker-killswitch.test.ts new file mode 100644 index 0000000000..f39454ea38 --- /dev/null +++ b/packages/cli/test/catchup-runner-worker-killswitch.test.ts @@ -0,0 +1,197 @@ +// Pins the operator kill-switch for the issue #2006 progressive walk. +// +// `DKG_CATCHUP_STOP_ON_PROOF=0` must restore the PREVIOUS behaviour exactly: +// every sync-capable peer contacted, both requested planes pulled from each of +// them, and no early stop — so an operator can back the optimisation out +// without a redeploy if a graph ever lands short. The switch is read once at +// module load, which is why this lives in its own file. +import { afterAll, describe, expect, it, vi } from 'vitest'; +import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS } from '@origintrail-official/dkg-agent'; +import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; + +// `vi.hoisted` runs before imports so the module-load-time constant picks this +// up — but it mutates the REAL process env, and vitest can reuse a worker +// process across files in a shard. Anything loaded afterwards, including a +// daemon spawned by a sibling suite, would otherwise inherit the kill-switch. +const previousCATCHUPSTOPONPROOF = vi.hoisted(() => { + const before = process.env.DKG_CATCHUP_STOP_ON_PROOF; + process.env.DKG_CATCHUP_STOP_ON_PROOF = '0'; + return before; +}); + +afterAll(() => { + if (previousCATCHUPSTOPONPROOF === undefined) delete process.env.DKG_CATCHUP_STOP_ON_PROOF; + else process.env.DKG_CATCHUP_STOP_ON_PROOF = previousCATCHUPSTOPONPROOF; +}); + +const fakeParentPort = vi.hoisted(() => { + const messageListeners: Array<(message: any) => void> = []; + const port = { + on(event: string, listener: (message: any) => void) { + if (event === 'message') messageListeners.push(listener); + }, + onPosted: undefined as ((message: any) => void) | undefined, + postMessage(message: any) { + port.onPosted?.(message); + }, + emitMessage(message: any) { + for (const listener of messageListeners) listener(message); + }, + }; + return port; +}); + +vi.mock('node:worker_threads', async (importOriginal) => ({ + ...(await importOriginal()), + parentPort: fakeParentPort, +})); + +const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +function durableResult() { + return { + insertedTriples: 1, + complete: true, + fetchedMetaTriples: 0, + fetchedDataTriples: 1, + insertedMetaTriples: 0, + insertedDataTriples: 1, + bytesReceived: 10, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 1, + checkpointAdvances: 0, + emptyResponses: 0, + metaOnlyResponses: 0, + dataRejectedMissingMeta: 0, + rejectedKcs: 0, + failedPeers: 0, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + +function sharedResult() { + return { + insertedTriples: 1, + fetchedMetaTriples: 0, + fetchedDataTriples: 1, + insertedMetaTriples: 0, + insertedDataTriples: 1, + bytesReceived: 10, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 1, + checkpointAdvances: 0, + emptyResponses: 0, + droppedDataTriples: 0, + failedPeers: 0, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + +let nextRunId = 1; + +async function runWorkerCatchup( + request: CatchupRunRequest, + handler: (method: string, args: unknown[]) => Promise, +): Promise { + await import('../src/catchup-runner-worker-impl.js'); + const runId = nextRunId++; + return new Promise((resolve, reject) => { + fakeParentPort.onPosted = (message: any) => { + if (message.type === 'invoke') { + handler(message.method, message.args).then( + (result) => fakeParentPort.emitMessage({ type: 'invoke-result', invokeId: message.invokeId, result }), + (error: unknown) => fakeParentPort.emitMessage({ + type: 'invoke-result', + invokeId: message.invokeId, + error: error instanceof Error ? error.message : String(error), + }), + ); + return; + } + if (message.type === 'run-result' && message.runId === runId) { + if (message.error) reject(new Error(message.error)); + else resolve(message.result as CatchupJobResult); + } + }; + fakeParentPort.emitMessage({ type: 'run', runId, request }); + }); +} + +/** + * SCOPE OF THESE TESTS — read before trusting a green run. + * + * Every case below hands the worker an `authoritativePeerId` through the stubbed + * `prepareCatchup` boundary. **No production resolver route currently produces + * one.** `resolveCuratorSyncPeer` was changed in `e7f46dca2` so that nothing + * earns `metadata` provenance, because a curator-to-peer binding read out of + * accumulated `/_meta` identifies the graph that HOLDS the rows, not the + * writer that SUPPLIED them — and ordinary durable-meta catch-up lets a + * contacted peer write those very rows. + * + * So these tests verify that the worker HANDLES an authority correctly IF it is + * given one. They do NOT verify that the early stop or the per-plane narrowing + * happens in the shipped build — it cannot, and byte volume is at the pre-fix + * level until #2018 lands a trusted binding. Read as end-to-end evidence for the + * fan-out reduction they would be claiming something untrue. + * + * They are kept rather than deleted because #2018 re-enables exactly this + * machinery, and deleting them would remove the contract it has to satisfy. When + * that lands, the missing piece is a case that derives `authoritativePeerId` + * through the REAL resolver/projection path instead of injecting it here. + */ +describe('catch-up progressive walk kill-switch', () => { + it('restores the full fan-out over every peer and every requested plane', async () => { + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + let inFlight = 0; + let peak = 0; + + // The curator is first AND cleanly proves both planes on the very first + // peer — with the switch ON this run would stop after `peer-0`. + const result = await runWorkerCatchup({ contextGraphId: 'cg-killswitch', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(2); + inFlight -= 1; + return durableResult(); + } + case 'syncSharedMemory': { + sharedCalls.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(2); + inFlight -= 1; + return sharedResult(); + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(peerIds); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersTried).toBe(peerIds.length); + expect(result.peersNotAttempted).toBe(0); + expect(result.dataSynced).toBe(peerIds.length); + expect(result.sharedMemorySynced).toBe(peerIds.length); + // The pre-existing sync-storm bound still applies with the switch off. + expect(peak).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(peak).toBeGreaterThan(1); + }); +}); diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts new file mode 100644 index 0000000000..ab85cc427f --- /dev/null +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -0,0 +1,334 @@ +// The daemon's subscribe job awaits `catchupRunner.run(...)` in a +// fire-and-forget async IIFE with no timeout of its own. `close()` terminates +// the Worker, and a terminated worker emits 'exit' — never 'error' — so before +// this fix a pending run promise was simply never settled and the job stayed +// `running` with no `finishedAt` for the rest of the process's life. Issue #2006 +// makes that reachable routinely, because a walk can be in flight much longer. +// +// `node:worker_threads` is mocked with a minimal fake Worker rather than +// spawning a real thread: nesting a real worker inside vitest's multi-file pool +// does not boot reliably, and the contract under test is this file's own +// handler wiring, not Node's (documented) 'exit'-on-terminate behaviour. +import { describe, expect, it, vi } from 'vitest'; +import type { DKGAgent } from '@origintrail-official/dkg-agent'; + +type Listener = (...args: unknown[]) => void; + +const workerControl = vi.hoisted(() => { + const state = { + listeners: new Map(), + posted: [] as unknown[], + terminated: false, + }; + class FakeWorker { + constructor(_path: string) { + state.listeners.clear(); + state.posted.length = 0; + state.terminated = false; + } + + on(event: string, listener: Listener) { + const existing = state.listeners.get(event) ?? []; + existing.push(listener); + state.listeners.set(event, existing); + } + + /** Test-only: deliver a message as if the worker had sent it. */ + static emitToRunner(message: unknown) { + for (const listener of state.listeners.get('message') ?? []) listener(message); + } + + /** Test-only: raise an arbitrary worker event (`error`, `exit`, …). */ + static emitEvent(event: string, ...args: unknown[]) { + for (const listener of state.listeners.get(event) ?? []) listener(...args); + } + + postMessage(message: unknown) { + state.posted.push(message); + } + + async terminate() { + state.terminated = true; + // Node emits 'exit' for a terminated worker, with code 1. It does NOT + // emit 'error' — which is exactly why the 'exit' handler is needed. + for (const listener of state.listeners.get('exit') ?? []) listener(1); + return 1; + } + } + return { state, FakeWorker }; +}); + +vi.mock('node:worker_threads', async (importOriginal) => ({ + ...(await importOriginal()), + Worker: workerControl.FakeWorker, +})); + +const { createCatchupRunner } = await import('../src/catchup-runner.js'); + +const stubAgent = {} as unknown as DKGAgent; + +/** Fail fast on a regression: an unsettled run must not burn the test timeout. */ +function withinTick(promise: Promise): Promise { + return Promise.race([ + promise, + new Promise<'still-pending'>((resolve) => { setTimeout(() => resolve('still-pending'), 500); }), + ]); +} + +describe('WorkerCatchupRunner lifecycle', () => { + it('rejects an in-flight run when the worker exits instead of leaving it pending forever', async () => { + const runner = createCatchupRunner(stubAgent); + const run = runner.run({ contextGraphId: 'cg-hang', includeSharedMemory: false }); + const settled = run.then(() => 'resolved' as const, (error: Error) => error); + + // The run was dispatched and is awaiting a `run-result` that will never come. + expect(workerControl.state.posted).toHaveLength(1); + expect(workerControl.state.posted[0]).toMatchObject({ type: 'run' }); + + await runner.close(); + + const outcome = await withinTick(settled); + expect(outcome).toBeInstanceOf(Error); + expect((outcome as Error).message).toContain('exited'); + }); + + it('fails a run started after the worker died instead of posting into the void', async () => { + // The pending-run case alone is not enough: the runner is constructed once + // per daemon, and `postMessage` to a dead worker neither throws nor + // delivers, so without the latch every LATER subscribe hung too. + const runner = createCatchupRunner(stubAgent); + await runner.close(); + const postedBeforeLaterRun = workerControl.state.posted.length; + + const later = runner.run({ contextGraphId: 'cg-later', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + + const outcome = await withinTick(later); + expect(outcome).toBeInstanceOf(Error); + expect((outcome as Error).message).toContain('exited'); + // …and it must not have queued work onto the dead worker. + expect(workerControl.state.posted).toHaveLength(postedBeforeLaterRun); + }); + + it('latches a worker `error` for the in-flight run AND every later one', async () => { + // Node usually emits `exit` after `error`, so the exit tests cover many real + // crashes indirectly — but the `error` handler moved from a one-off pending + // rejection to the shared latch, and nothing pinned that. Restoring the old + // behaviour would let a LATER subscribe post into a dead worker again, which + // is the half of #2006's hang that made every subsequent job stick at + // `running`. + const runner = createCatchupRunner(stubAgent); + const inFlight = runner.run({ contextGraphId: 'cg-crash', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + expect(workerControl.state.posted).toHaveLength(1); + + workerControl.FakeWorker.emitEvent('error', new Error('boom')); + + const first = await withinTick(inFlight); + expect(first).toBeInstanceOf(Error); + expect((first as Error).message).toContain('boom'); + + // The latch, not just this run: a later run must fail fast… + const postedBeforeLater = workerControl.state.posted.length; + const later = runner.run({ contextGraphId: 'cg-after-crash', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + const second = await withinTick(later); + expect(second).toBeInstanceOf(Error); + // …and must not have queued work onto the dead worker. + expect(workerControl.state.posted).toHaveLength(postedBeforeLater); + }); + + it('rejects every pending run exactly once', async () => { + const runner = createCatchupRunner(stubAgent); + const first = runner.run({ contextGraphId: 'cg-a', includeSharedMemory: false }) + .then(() => 'resolved' as const, () => 'rejected' as const); + const second = runner.run({ contextGraphId: 'cg-b', includeSharedMemory: true }) + .then(() => 'resolved' as const, () => 'rejected' as const); + + await runner.close(); + // A second close (or a late 'error' after 'exit') must not double-settle. + await runner.close(); + + await expect(withinTick(Promise.all([first, second]))) + .resolves.toEqual(['rejected', 'rejected']); + }); +}); + +/** + * The parent-side bridge (`WorkerCatchupRunner.invokeAgent`) is the boundary + * every worker-impl test stubs out — those tests supply `authoritativePeerId` + * and observe `source` themselves, so a regression HERE would leave them green + * while production early-stopped on a stale bootstrap hint or reported + * foreground catch-up as `unspecified` in scheduler diagnostics. + * + * NOTE on the authoritative cases below: they stub `resolveSyncPeerWithProvenance` + * to return `provenance: 'metadata'`, which the real resolver no longer returns + * for ANY input (`e7f46dca2`). They pin that the bridge PROPAGATES an authority + * faithfully — that `authoritativeSyncPeerId` is the single definition of who + * may end a walk, and that the bridge does not invent one from a bootstrap hint. + * They are not evidence that an authority is ever produced. See #2018. + */ +describe('WorkerCatchupRunner agent bridge', () => { + function bridgeAgent(overrides: Record = {}) { + const calls: Record = { durable: [], shared: [] }; + let resolutionCalls = 0; + const agent = { + isPrivateContextGraph: async () => false, + resolveSyncPeerWithProvenance: async () => { + resolutionCalls += 1; + return { peerId: 'peer-hint', provenance: 'bootstrap-hint' }; + }, + ensurePeerConnected: async () => {}, + primeCatchupConnections: async () => {}, + selectCatchupPeers: (peers: Array<{ toString(): string }>) => peers, + node: { libp2p: { getConnections: () => [] } }, + syncFromPeerDetailed: async (...args: unknown[]) => { + calls.durable.push(args); + return {}; + }, + syncSharedMemoryFromPeerDetailed: async (...args: unknown[]) => { + calls.shared.push(args); + return {}; + }, + ...overrides, + }; + return { + agent: agent as unknown as DKGAgent, + calls, + resolutionCalls: () => resolutionCalls, + }; + } + + /** Drive one `invoke` through the real bridge and return what it posted back. */ + async function invokeThroughBridge(agent: DKGAgent, method: string, args: unknown[]) { + createCatchupRunner(agent); + const before = workerControl.state.posted.length; + workerControl.FakeWorker.emitToRunner({ type: 'invoke', invokeId: 1, method, args }); + for (let i = 0; i < 50 && workerControl.state.posted.length === before; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 1)); + } + return workerControl.state.posted.at(-1) as { result?: any; error?: string }; + } + + it('does not report a bootstrap-hint peer as the catch-up authority', async () => { + const { agent, resolutionCalls } = bridgeAgent(); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-hint']); + + // The hint still ranks the walk… + expect(posted.result.preferredPeerId).toBe('peer-hint'); + // …but must not be handed to the worker as an authority. + expect(posted.result.authoritativePeerId).toBeUndefined(); + // …and both notions came from ONE resolution. The resolver reads `_meta` + // (and may hit the agent registry), and it evicts the bootstrap hint once + // metadata confirms a curator — so a second call is neither free nor the + // same call. + expect(resolutionCalls()).toBe(1); + }); + + it('reports a metadata-resolved curator as the catch-up authority', async () => { + const { agent, resolutionCalls } = bridgeAgent({ + resolveSyncPeerWithProvenance: async () => ({ + peerId: 'peer-curator', + provenance: 'metadata', + }), + }); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-meta']); + + expect(posted.result.preferredPeerId).toBe('peer-curator'); + expect(posted.result.authoritativePeerId).toBe('peer-curator'); + expect(resolutionCalls()).toBe(0); + }); + + it('falls back to ranking alone when the agent predates the provenance resolver', async () => { + // The bridge talks to whatever agent the daemon composed; one without + // `resolveSyncPeerWithProvenance` must still rank the walk rather than + // throw — and must NOT infer an authority it cannot establish. + const { agent } = bridgeAgent({ + resolveSyncPeerWithProvenance: undefined, + resolvePreferredSyncPeerId: async () => 'peer-legacy', + }); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-legacy']); + + expect(posted.result.preferredPeerId).toBe('peer-legacy'); + expect(posted.result.authoritativePeerId).toBeUndefined(); + }); + + it('clamps an unbounded RPC source at the untrusted edge', async () => { + // A worker RPC argument crossed a structured-clone boundary, so its + // compile-time type guaranteed nothing. If an identifier-bearing value + // reached the scheduler it would become a metric and log DIMENSION, + // re-opening the correlation-identifier leak that collapsing the operation + // label was added to close, and multiplying diagnostic cardinality. + const { agent, calls } = bridgeAgent(); + + await invokeThroughBridge(agent, 'syncDurable', ['peer-a', 'cg-x', 2000, 'durable:urn:cg:private:abc']); + await invokeThroughBridge(agent, 'syncSharedMemory', ['peer-a', 'cg-x', 2000, { not: 'a string' }]); + + expect(calls.durable[0]!.at(-1)).toMatchObject({ source: 'unspecified' }); + expect(calls.shared[0]!.at(-1)).toMatchObject({ source: 'unspecified' }); + }); + + it('hands the resolved peer into selection and returns an authority-ranked list', async () => { + // The whole load reduction depends on the walk opening with the curator, + // and the worker only ever sees an ALREADY-ranked `peerIds`. Every + // worker test supplies that list itself, so the handoff that produces it — + // resolve, then rank the live connections against the resolved peer — was + // covered nowhere. Dropping the second argument here would leave the worker + // suite green while production opened at the full cap. + const selectCalls: unknown[][] = []; + const { agent } = bridgeAgent({ + resolveSyncPeerWithProvenance: async () => ({ + peerId: 'peer-curator', + provenance: 'metadata', + }), + node: { + libp2p: { + getConnections: () => ['peer-b', 'peer-curator', 'peer-a', 'peer-b'].map( + (id) => ({ remotePeer: { toString: () => id } }), + ), + }, + }, + selectCatchupPeers: (...args: unknown[]) => { + selectCalls.push(args); + const peers = args[0] as Array<{ toString(): string }>; + const preferred = args[1] as string | undefined; + // Stand in for the real ranking: preferred first, rest in order. + return [...peers].sort((a, b) => Number(b.toString() === preferred) + - Number(a.toString() === preferred)); + }, + }); + + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-rank']); + + expect(selectCalls).toHaveLength(1); + const [candidates, preferred, isPrivate] = selectCalls[0]!; + // Live connections are de-duplicated before ranking… + expect((candidates as Array<{ toString(): string }>).map((p) => p.toString())) + .toEqual(['peer-b', 'peer-curator', 'peer-a']); + // …the RESOLVED peer is what selection ranks against… + expect(preferred).toBe('peer-curator'); + expect(isPrivate).toBe(false); + // …and the worker receives the ranked ids as plain strings, curator first. + expect(posted.result.peerIds).toEqual(['peer-curator', 'peer-b', 'peer-a']); + expect(posted.result.authoritativePeerId).toBe('peer-curator'); + expect(posted.result.connectedPeers).toBe(3); + }); + + it('forwards the admission source into both detailed sync calls', async () => { + const { agent, calls } = bridgeAgent(); + + await invokeThroughBridge(agent, 'syncDurable', ['peer-a', 'cg-x', 2000, 'catchup-foreground']); + await invokeThroughBridge(agent, 'syncSharedMemory', ['peer-a', 'cg-x', 2000, 'catchup-foreground']); + + expect(calls.durable).toHaveLength(1); + expect(calls.durable[0]!.at(-1)).toMatchObject({ + priority: 2000, + source: 'catchup-foreground', + }); + expect(calls.shared).toHaveLength(1); + expect(calls.shared[0]!.at(-1)).toMatchObject({ + priority: 2000, + source: 'catchup-foreground', + }); + }); +}); diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index adf7558284..e4eacd6eff 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -1,8 +1,13 @@ import { describe, expect, it, vi } from 'vitest'; import { + catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByAuthorityHostedEmpty, + catchupPlaneProvenByData, + catchupPlaneProvenByUnanimousEmpty, + catchupPlaneReady, classifyDurableCatchupRequest, runDurableCatchupLeg, } from '../src/catchup-runner.js'; @@ -635,3 +640,394 @@ describe('route-level durable catchup orchestration', () => { }); }); }); + +// Issue #2006. On the wire, a peer that hosts an empty Context Graph and a peer +// that has never heard of it are byte-identical: an unknown CG has no access +// policy, so the responder authorizes the request and its CG-scoped queries +// simply return zero rows. The requester emits `emptyResponses` only when BOTH +// phase payloads are empty, so an empty response can never carry hosting +// evidence. Emptiness is therefore only provable as a whole-round verdict. +describe('catch-up plane proof predicates', () => { + const noEvidence = { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }; + const cleanEmptyRound = { + fetchedMetaTriples: 0, + fetchedDataTriples: 0, + emptyResponses: 2, + failedPeers: 0, + failedPhases: 0, + timedOutPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; + const emptyPeers = { ...noEvidence, emptyPeers: 2 }; + + it('treats verified data and verified private-only completions as positive proof', () => { + expect(catchupPlaneProvenByData({ ...noEvidence, verifiedDataPeers: 1 })).toBe(true); + expect(catchupPlaneProvenByData({ ...noEvidence, verifiedPrivateOnlyPeers: 1 })).toBe(true); + expect(catchupPlaneProvenByData(noEvidence)).toBe(false); + expect(catchupPlaneProvenByData(undefined)).toBe(false); + // A clean empty response is NOT positive proof — it can never stop the walk. + expect(catchupPlaneProvenByData(emptyPeers)).toBe(false); + }); + + it('accepts a unanimously clean, content-free public round as proof of emptiness', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, cleanEmptyRound, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, cleanEmptyRound, { isPrivate: false })).toBe(true); + }); + + it('never proves a private plane from an empty round', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, cleanEmptyRound, { isPrivate: true })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, cleanEmptyRound, { isPrivate: true })).toBe(false); + }); + + it.each([ + ['a data-bearing peer that failed', { fetchedDataTriples: 122_705, failedPhases: 5 }], + ['fetched data with no verified completion', { fetchedDataTriples: 5_000 }], + ['a failed phase', { failedPhases: 1 }], + ['a timed-out phase', { timedOutPhases: 1 }], + ['a denial', { deniedPhases: 1 }], + ['a local admission deferral', { deferredBackpressure: 1 }], + // An integrity rejection is stronger than a failure: it is a peer that + // SERVED CONTENT for this graph which then failed verification, so it is + // positive evidence the graph is not empty. `classifyDurableProgress` + // already treats both as blocking failures per peer. + ['data rejected for missing metadata', { dataRejectedMissingMeta: 1 }], + ['a rejected Knowledge Collection', { rejectedKcs: 1 }], + ])('voids the empty proof when the round contains %s', (_label, overrides) => { + const diagnostics = { ...cleanEmptyRound, ...overrides }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(false); + }); + + it.each([ + // Every registered Context Graph carries definition triples in its own + // `/_meta`, so ANY peer that hosts the graph returns metadata even when + // the graph holds zero Knowledge Assets. Treating that as content would make + // a legitimately empty public graph permanently unreadable. + ['metadata from a hosting peer', { fetchedMetaTriples: 12 }], + // A transport failure is a peer we never heard from. On a live testnet a + // majority of connected peers can be unreachable; an unreachable stranger is + // evidence of nothing. A peer that DID engage and then failed shows up in + // the voiding counters above. + ['a transport failure to an unreachable peer', { failedPeers: 4 }], + ])('still proves an empty public round despite %s', (_label, overrides) => { + const diagnostics = { ...cleanEmptyRound, ...overrides }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(true); + }); + + it('proves a registered public graph that simply has no Knowledge Assets yet', () => { + // The shape a freshly registered, still-empty public Context Graph actually + // produces: its host serves the CG definition triples (metadata) and no + // data, other peers answer clean-empty, and some connected peers are + // unreachable. This must reach `done`, not sit at `unreachable` forever. + const registeredButEmpty = { + ...cleanEmptyRound, + fetchedMetaTriples: 9, + emptyResponses: 3, + failedPeers: 2, + }; + expect(catchupPlaneReady(emptyPeers, registeredButEmpty, { isPrivate: false })).toBe(true); + }); + + it('still reports ready when a peer delivered verified data despite other failures', () => { + const diagnostics = { ...cleanEmptyRound, fetchedDataTriples: 24_541, failedPhases: 1 }; + const completion = { ...emptyPeers, verifiedDataPeers: 1 }; + expect(catchupPlaneReady(completion, diagnostics, { isPrivate: false })).toBe(true); + expect(catchupPlaneReady(completion, diagnostics, { isPrivate: true })).toBe(true); + }); + + it('requires at least one clean empty completion before an empty verdict', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + noEvidence, + { ...cleanEmptyRound, emptyResponses: 0 }, + { isPrivate: false }, + )).toBe(false); + }); + + // A registered public graph that really is empty still carries definition + // triples in its own `/_meta`, so the peer hosting it answers + // metadata-only, never wire-empty. Nothing in the whole-round rule above can + // ever fire for it — the curator has to say so itself. + describe('an empty graph whose only responder is its curator', () => { + const hostedEmptyRound = { + insertedTriples: 9, + insertedMetaTriples: 9, + insertedDataTriples: 0, + fetchedDataTriples: 0, + metaOnlyResponses: 1, + emptyResponses: 0, + completedPhases: 2, + }; + const hostedEmptyDiagnostics = { + ...cleanEmptyRound, + fetchedMetaTriples: 9, + emptyResponses: 0, + }; + + it('never reads a SHARED-MEMORY round as hosted-empty evidence', () => { + // `/_meta` definition triples are a DURABLE fact: serving them proves + // the peer hosts the Context Graph. Shared-memory metadata is a different + // artifact, and shared memory is contributed by many members rather than + // owned by the curator — so "the curator has SWM structure but no SWM + // rows" does not mean the network has none. Treating it as hosted-empty + // would settle the shared plane and stop the walk before any member that + // actually holds the SWM data is contacted. + const curatorSharedMetaOnly = { + insertedTriples: 5, + insertedMetaTriples: 5, + insertedDataTriples: 0, + fetchedDataTriples: 0, + emptyResponses: 0, + completedPhases: 2, + }; + + expect(catchupPeerPlaneEvidence(curatorSharedMetaOnly, { + fromAuthority: true, + plane: 'shared-memory', + })).toMatchObject({ authorityEmptyPeers: 0 }); + + // Nor does a wire-empty one: on this plane NOBODY's emptiness is + // authoritative, because the curator does not own the members' layers. + // An empty SWM plane is still provable, but only as a whole-round verdict. + expect(catchupPeerPlaneEvidence( + { ...curatorSharedMetaOnly, insertedTriples: 0, insertedMetaTriples: 0, emptyResponses: 1 }, + { fromAuthority: true, plane: 'shared-memory' }, + )).toMatchObject({ authorityEmptyPeers: 0, emptyPeers: 1 }); + + // …and the identical shape on the DURABLE plane is hosting evidence. + expect(catchupPeerPlaneEvidence(curatorSharedMetaOnly, { + complete: true, + fromAuthority: true, + plane: 'durable', + })).toMatchObject({ authorityEmptyPeers: 1 }); + }); + + it('counts the curator, and ONLY the curator, as hosted-empty evidence', () => { + expect(catchupPeerPlaneEvidence(hostedEmptyRound, { + plane: 'durable', + complete: true, + fromAuthority: true, + })).toMatchObject({ verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 1 }); + // The identical round from any other peer is the commonest state on the + // network — a member holding `_meta` that has not synced the data yet — + // and counting it would resettle #2006 as `done` with zero KAs. + expect(catchupPeerPlaneEvidence(hostedEmptyRound, { plane: 'durable', complete: true })) + .toMatchObject({ authorityEmptyPeers: 0 }); + // Neither does a curator round that fetched data but inserted none. + expect(catchupPeerPlaneEvidence( + { ...hostedEmptyRound, fetchedDataTriples: 4_000 }, + { plane: 'durable', complete: true, fromAuthority: true }, + )).toMatchObject({ authorityEmptyPeers: 0 }); + }); + + it('proves the public plane with no wire-empty response anywhere in the round', () => { + const completion = { ...noEvidence, authorityEmptyPeers: 1 }; + expect(catchupPlaneProvenByAuthorityHostedEmpty( + completion, + hostedEmptyDiagnostics, + { isPrivate: false }, + )).toBe(true); + expect(catchupPlaneReady(completion, hostedEmptyDiagnostics, { isPrivate: false })).toBe(true); + // Without the curator's own evidence the same round proves nothing. + expect(catchupPlaneReady(noEvidence, hostedEmptyDiagnostics, { isPrivate: false })).toBe(false); + }); + + it('is voided when another peer delivered data the curator did not have', () => { + expect(catchupPlaneProvenByAuthorityHostedEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + { ...hostedEmptyDiagnostics, fetchedDataTriples: 122_705 }, + { isPrivate: false }, + )).toBe(false); + }); + + it.each([ + ['data rejected for missing metadata', { dataRejectedMissingMeta: 1 }], + ['a rejected Knowledge Collection', { rejectedKcs: 1 }], + ])('is voided by %s elsewhere in the round, ahead of the curator\'s word', (_label, overrides) => { + // Content that failed verification still proves content EXISTS, which + // outranks the curator saying the graph is empty — unlike a plain + // transport or phase failure, which the curator's answer does outrank. + expect(catchupPlaneProvenByAuthorityHostedEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + { ...hostedEmptyDiagnostics, ...overrides }, + { isPrivate: false }, + )).toBe(false); + }); + + it('never proves a private plane', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + hostedEmptyDiagnostics, + { isPrivate: true }, + )).toBe(false); + }); + }); + + describe('a non-curator that has `_meta` but no data', () => { + // The requester itself logs "peer may have empty or pruned data graph" for + // this response, which names the ambiguity exactly: the graph is empty, OR + // this member has not synced it yet. Without the curator present there is + // nothing to resolve it against, and combining it with an unrelated peer's + // empty answer would settle a 40-KA graph as `done` with zero. + const memberWithMetaOnly = { + ...cleanEmptyRound, + emptyResponses: 1, + metaOnlyResponses: 1, + fetchedMetaTriples: 9, + }; + + it('cannot be combined with a stranger\'s empty answer to prove the plane', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(false); + expect(catchupPlaneReady( + { ...noEvidence, emptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(false); + }); + + it('costs the legitimately empty graph nothing once its CURATOR answers', () => { + // The positive half. Voiding on `metaOnlyResponses` would be a bad trade + // if it also blocked the real empty-public-graph case — it does not, + // because the curator's own round settles that through the other proof + // mode, which is evaluated independently. + expect(catchupPlaneReady( + { ...noEvidence, emptyPeers: 1, authorityEmptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(true); + }); + + it('leaves the all-strangers round provable, so the rule is not vacuous', () => { + // A tightened clause that can never be satisfied is worse than no clause, + // because nothing reveals it. Pin that the unanimous rule still fires + // when every responder answered wire-empty and nobody returned metadata. + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 2 }, + { ...cleanEmptyRound, metaOnlyResponses: 0 }, + { isPrivate: false }, + )).toBe(true); + }); + }); + + describe('a curator that was selected but never cleanly answered', () => { + // The walk puts a resolvable curator ALONE in wave 1, so when it + // transport-fails the walk moves on to strangers, one answers empty, and the + // graph's 40 Knowledge Assets get reported as zero. That is issue #2006's own + // symptom in its sharpest form. + const curatorSilent = { ...cleanEmptyRound, failedPeers: 1, authorityUnanswered: true }; + + it('cannot have its plane proven by a stranger answering empty', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, curatorSilent, { isPrivate: false })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, curatorSilent, { isPrivate: false })).toBe(false); + }); + + it.each([ + // No curator resolved at all. The hosted-empty backstop structurally + // cannot fire here, so voiding on a mere unreachable STRANGER would pin a + // legitimately empty public graph at `unreachable` forever — the liveness + // failure this rule exists to avoid. Only the CURATOR's silence is decisive. + ['no curator was resolvable', { ...cleanEmptyRound, failedPeers: 4 }], + // Registered-but-empty public graph on a lossy network, curator absent + // from the round entirely. + ['the graph is registered but empty', { + ...cleanEmptyRound, fetchedMetaTriples: 9, emptyResponses: 3, failedPeers: 2, + }], + ])('still proves an empty round when %s', (_label, diagnostics) => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(true); + }); + + it('is still proven when the curator DID answer, unreachable strangers aside', () => { + // The positive complement: the flag is about the curator's silence, not + // about the round being lossy. + const curatorAnswered = { ...cleanEmptyRound, failedPeers: 3, authorityUnanswered: false }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, curatorAnswered, { isPrivate: false })) + .toBe(true); + }); + }); + + it('uses per-peer completion evidence, and the aggregate ONLY without it', () => { + // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter + // (`diagnostics.emptyResponses`) are separate carriers, but they are not + // interchangeable and must not be ORed together. + // + // `emptyResponses` counts an empty PAYLOAD; `emptyPeers` counts a peer whose + // round was empty AND clean. A peer that answered empty but did not complete + // raises the first and not the second — so consulting the aggregate when + // per-peer evidence exists lets an explicitly incomplete response prove the + // plane ready, which is the false-`done` class this proof exists to prevent. + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 1 }, + { ...cleanEmptyRound, emptyResponses: 0 }, + { isPrivate: false }, + )).toBe(true); + + // Completion evidence PRESENT and negative: the aggregate must not re-admit + // it. This is the assertion that fails if the carriers are ORed. + expect(catchupPlaneProvenByUnanimousEmpty( + noEvidence, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, + )).toBe(false); + + // Completion evidence genuinely ABSENT (the legacy runner result): the + // aggregate is the only carrier there is, so it still counts. Without this + // row, dropping the fallback entirely would look like a passing change. + expect(catchupPlaneProvenByUnanimousEmpty( + undefined, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, + )).toBe(true); + }); + + it('does not let an explicitly incomplete empty peer prove the plane', () => { + // The production shape of the row above: the worker reports a peer that + // returned an empty payload but whose round never completed, so the peer is + // absent from `emptyPeers` while `emptyResponses` still counts it. + const incompleteEmpty = catchupPeerPlaneEvidence( + { emptyResponses: 1, completedPhases: 0, bytesReceived: 0 }, + { plane: 'durable', complete: false }, + ); + expect(incompleteEmpty.emptyPeers).toBe(0); + expect(catchupPlaneProvenByUnanimousEmpty( + incompleteEmpty, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, + )).toBe(false); + }); +}); + +describe('catch-up peer accounting with a skipped plane', () => { + const cleanShared = { + insertedTriples: 3, + insertedDataTriples: 3, + completedPhases: 1, + bytesReceived: 30, + }; + + it('does not read a skipped durable plane as a peer response', () => { + // The walk omits the durable plane for peers contacted purely as a + // shared-memory fallback. An absent plane is not a silent one: it must not + // manufacture a response for a peer whose only requested plane failed. + expect(catchupPeerResponded(null, { failedPeers: 1 })).toBe(false); + expect(catchupPeerResponded(null, undefined)).toBe(false); + expect(catchupPeerResponded(null, cleanShared)).toBe(true); + }); + + it('judges a skipped durable plane purely on the shared-memory outcome', () => { + expect(catchupPeerSucceeded(null, cleanShared, false)).toBe(true); + expect(catchupPeerSucceeded(null, { ...cleanShared, timedOutPhases: 1 }, false)).toBe(false); + expect(catchupPeerSucceeded(null, { failedPeers: 1 }, false)).toBe(false); + }); +}); diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 3378796855..8071901332 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from 'vitest'; import type { CatchupJobResult } from '../src/catchup-runner.js'; -import { classifyContextGraphCatchupReadiness } from '../src/context-graph-readiness.js'; +import { + CONTEXT_GRAPH_READINESS_VERSION, + classifyContextGraphCatchupReadiness, +} from '../src/context-graph-readiness.js'; function mixedPeerResult(verifiedDataPeers: number): CatchupJobResult { return { @@ -186,20 +189,32 @@ describe('context graph catch-up readiness classification', () => { expect(classification.eventPayload).toBeUndefined(); }); - it('accepts a public clean-empty peer when another peer denies', () => { + // Emptiness is only provable as a whole-round verdict: an empty response is + // byte-identical whether the peer hosts an empty graph or has never heard of + // it, so a clean-empty peer proves the plane only when NOBODY in the round + // delivered content and nothing failed. + function publicEmptyRoundResult(): CatchupJobResult { const result = mixedPeerResult(0); result.dataSynced = 0; - result.peersSucceeded = 1; + result.peersSucceeded = 2; + result.denied = false; + result.deniedPeers = 0; if (!result.cleanPlaneCompletions || !result.diagnostics?.durable) { throw new Error('durable completion evidence missing'); } - result.cleanPlaneCompletions.durable.emptyPeers = 1; + result.cleanPlaneCompletions.durable.emptyPeers = 2; result.diagnostics.durable.fetchedDataTriples = 0; result.diagnostics.durable.insertedDataTriples = 0; - result.diagnostics.durable.emptyResponses = 1; + result.diagnostics.durable.emptyResponses = 2; + result.diagnostics.durable.timedOutPhases = 0; + result.diagnostics.durable.deniedPhases = 0; + result.diagnostics.durable.completedPhases = 4; + return result; + } + it('persists a unanimously clean-empty round only because it was FULLY accounted', () => { const classification = classifyContextGraphCatchupReadiness({ - result, + result: publicEmptyRoundResult(), includeSharedMemory: false, hasConfirmedMeta: true, isPrivate: false, @@ -212,10 +227,307 @@ describe('context graph catch-up readiness classification', () => { synced: true, sharedMemorySynced: false, }, + // Every attempted peer answered (`failedPeers: 0`), so the empty verdict + // was taken over the whole peer set and may be written down. The same + // round with a peer unaccounted for must NOT be — see the next case. readinessPatch: { durableVerified: true, sharedMemoryVerified: false, }, }); + + // `synced` is a second persisted readiness bit and gates write preflight, + // so it has to carry the SAME verdict as the provenance patch. + expect(classification.statePatch?.synced) + .toBe(classification.readinessPatch?.durableVerified); + }); + + it('does not settle a round where one peer answered empty but never completed', () => { + // The shape the fully-accounted check alone cannot see. Peer A completes + // empty; peer B returns an empty payload but `complete: false`. + // `catchupPeerPlaneEvidence` erases B to an all-zero record, so `emptyPeers` + // stays 1 and — because an incomplete round is NOT a transport failure — + // `failedPeers` stays 0. Both the unanimous-empty proof and the + // fully-accounted gate would therefore pass, and the graph would be frozen + // as synced on half an answer. + const mixed = publicEmptyRoundResult(); + mixed.cleanPlaneCompletions!.durable.emptyPeers = 1; + mixed.cleanPlaneCompletions!.durable.incompleteResponders = 1; + mixed.diagnostics!.durable.emptyResponses = 2; + mixed.diagnostics!.durable.failedPeers = 0; + + const classification = classifyContextGraphCatchupReadiness({ + result: mixed, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + // …and nothing opens the writeability gate either. + expect(classification.statePatch?.synced).toBe(false); + }); + + it('re-derives an empty verdict instead of carrying it into the next run', () => { + // The false-`done` residual: with no authoritative curator, one unrelated + // empty response alongside transport-level peer failures satisfies the + // unanimous-empty proof. That is survivable as a per-run verdict, but the + // readiness OR against `readinessBeforeCatchup` would otherwise make it + // permanent — every later run would report `done` without re-proving + // anything, which is exactly the false-`done` class issue #2006 targets. + const firstRun = publicEmptyRoundResult(); + firstRun.cleanPlaneCompletions!.durable.emptyPeers = 1; + firstRun.diagnostics!.durable.emptyResponses = 1; + firstRun.diagnostics!.durable.failedPeers = 1; + + const first = classifyContextGraphCatchupReadiness({ + result: firstRun, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + expect(first.jobStatus).toBe('done'); + expect(first.readinessPatch).toMatchObject({ durableVerified: false }); + // …and the subscription must not be marked synced either, or write + // preflight (`contextGraphRowIsWritable`: `subscribed && synced`) would + // grant durable readiness the provenance store deliberately withheld. + expect(first.statePatch?.synced).toBe(false); + + // Second run: metadata only, nothing proven. Feed back exactly what run one + // persisted. If the empty verdict had been frozen, this would still say + // `done` while proving nothing. + const secondRun = publicEmptyRoundResult(); + secondRun.cleanPlaneCompletions!.durable.emptyPeers = 0; + secondRun.diagnostics!.durable.emptyResponses = 0; + secondRun.diagnostics!.durable.metaOnlyResponses = 1; + + const second = classifyContextGraphCatchupReadiness({ + result: secondRun, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup: { + ...readinessBeforeCatchup, + version: CONTEXT_GRAPH_READINESS_VERSION, + durableVerified: first.readinessPatch!.durableVerified!, + }, + }); + expect(second.jobStatus).not.toBe('done'); + + // The other half of the contract: readiness proven by CONTENT is still + // sticky, so a graph that really did sync does not re-prove itself forever. + const proven = publicEmptyRoundResult(); + proven.dataSynced = 12; + proven.cleanPlaneCompletions!.durable.verifiedDataPeers = 1; + expect(classifyContextGraphCatchupReadiness({ + result: proven, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }).readinessPatch).toMatchObject({ durableVerified: true }); + }); + + it('does not accept a public clean-empty peer when another peer denies', () => { + // A denial means we did not hear from every peer, so "nobody has anything" + // is not established. Before #2006 this returned `done`. + const result = publicEmptyRoundResult(); + result.denied = true; + result.deniedPeers = 1; + result.cleanPlaneCompletions!.durable.emptyPeers = 1; + result.diagnostics!.durable.emptyResponses = 1; + result.diagnostics!.durable.deniedPhases = 1; + + const classification = classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification).toMatchObject({ + jobStatus: 'unreachable', + readinessPatch: { durableVerified: false, sharedMemoryVerified: false }, + }); + }); + + it('does not let a clean-empty peer mask a data-bearing peer that failed', () => { + // The exact reported #2006 shape: 122,705 triples fetched, five phases + // failed, no verified data completion, and unrelated peers answering empty. + const result = publicEmptyRoundResult(); + result.cleanPlaneCompletions!.durable.emptyPeers = 1; + result.diagnostics!.durable.emptyResponses = 1; + result.diagnostics!.durable.fetchedDataTriples = 122_705; + result.diagnostics!.durable.failedPhases = 5; + + const classification = classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + }); + + it('applies the same fail-closed empty rule to a legacy runner result', () => { + // A result without `cleanPlaneCompletions` (an older in-process runner + // during a rolling upgrade) takes the compatibility branch. It must not be + // a way around the round-level guard. + const masked = publicEmptyRoundResult(); + delete masked.cleanPlaneCompletions; + masked.diagnostics!.durable.fetchedDataTriples = 122_705; + masked.diagnostics!.durable.failedPhases = 5; + + expect(classifyContextGraphCatchupReadiness({ + result: masked, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }).jobStatus).not.toBe('done'); + + // …and a legacy result from a genuinely empty round still settles. + const clean = publicEmptyRoundResult(); + delete clean.cleanPlaneCompletions; + expect(classifyContextGraphCatchupReadiness({ + result: clean, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + // Same rule as the non-legacy path: a fully accounted empty round is + // written down, a partial one is not. + readinessPatch: { durableVerified: true }, + }); + + // The partial-round half of that rule, on the legacy branch too. + const lossy = publicEmptyRoundResult(); + delete lossy.cleanPlaneCompletions; + lossy.diagnostics!.durable.failedPeers = 1; + expect(classifyContextGraphCatchupReadiness({ + result: lossy, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + readinessPatch: { durableVerified: false }, + statePatch: { synced: false }, + }); + }); + + it('never proves a private plane from an empty round', () => { + const classification = classifyContextGraphCatchupReadiness({ + result: publicEmptyRoundResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: true, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).toBe('unreachable'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + }); + + // A registered public graph with no Knowledge Assets yet. Its host serves the + // CG definition triples from `/_meta`, so it answers metadata-only rather + // than wire-empty and the whole-round rule above can never fire — no peer in + // the round produced an `emptyResponses`. The curator's own hosted-empty + // round is the only evidence such a graph can produce. + function curatorHostedEmptyResult(): CatchupJobResult { + const result = publicEmptyRoundResult(); + if (!result.cleanPlaneCompletions || !result.diagnostics?.durable) { + throw new Error('durable completion evidence missing'); + } + result.cleanPlaneCompletions.durable.emptyPeers = 0; + result.cleanPlaneCompletions.durable.authorityEmptyPeers = 1; + result.diagnostics.durable.emptyResponses = 0; + result.diagnostics.durable.metaOnlyResponses = 1; + result.diagnostics.durable.fetchedMetaTriples = 9; + result.diagnostics.durable.insertedMetaTriples = 9; + return result; + } + + it('settles a registered-but-empty public graph on the curator hosted-empty round', () => { + expect(classifyContextGraphCatchupReadiness({ + result: curatorHostedEmptyResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + statePatch: { synced: true }, + readinessPatch: { durableVerified: true }, + }); + }); + + it.each([ + ['the round came from members rather than the curator', (result: CatchupJobResult) => { + result.cleanPlaneCompletions!.durable.authorityEmptyPeers = 0; + }], + ['another peer delivered data the curator did not have', (result: CatchupJobResult) => { + result.diagnostics!.durable.fetchedDataTriples = 122_705; + }], + ])('keeps the same round unready when %s', (_label, mutate) => { + const result = curatorHostedEmptyResult(); + mutate(result); + + expect(classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'unreachable', + readinessPatch: { durableVerified: false }, + }); + }); + + it('is not discarded by the denial gate before readiness is evaluated', () => { + // `cleanCompletionHasResponse` gates the denial and no-response branches + // that run BEFORE `catchupPlaneReady` is consulted. A new evidence carrier + // missing from that gate is silently unreachable: the durable plane would + // be provably ready and the job would still return `denied`, because a + // shared-memory phase from some other peer was refused. + const result = curatorHostedEmptyResult(); + result.denied = true; + result.deniedPeers = 1; + result.diagnostics!.sharedMemory.deniedPhases = 1; + + expect(classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + readinessPatch: { durableVerified: true }, + }); + }); + + it('never settles a PRIVATE plane on a curator hosted-empty round', () => { + // Private planes stay proof-by-content only: an authorized-but-filtered + // response is indistinguishable from an empty one on this side of the wire. + expect(classifyContextGraphCatchupReadiness({ + result: curatorHostedEmptyResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: true, + readinessBeforeCatchup, + }).jobStatus).toBe('unreachable'); }); }); diff --git a/packages/cli/test/context-graph-subscribe-readiness.test.ts b/packages/cli/test/context-graph-subscribe-readiness.test.ts index 6eac7e0137..ad47313dd5 100644 --- a/packages/cli/test/context-graph-subscribe-readiness.test.ts +++ b/packages/cli/test/context-graph-subscribe-readiness.test.ts @@ -528,7 +528,11 @@ describe('context graph subscribe readiness requires authoritative metadata', () }); }); - it('keeps a public clean-empty peer valid when another peer denies', async () => { + // Issue #2006: an empty response cannot distinguish "hosts an empty graph" + // from "never heard of this graph", so a clean-empty peer only proves the + // plane when the whole round was content-free and failure-free. A denial or a + // failed data-bearing peer means we did not hear from everyone. + it('does not keep a public clean-empty peer valid when another peer denies', async () => { const mixed = cleanEmptyResult(); mixed.connectedPeers = 2; mixed.totalPeers = 2; @@ -553,19 +557,47 @@ describe('context graph subscribe readiness requires authoritative metadata', () }, }); - expect(result.job.status).toBe('done'); - expect(result.job.error).toBeUndefined(); - expect(result.state).toMatchObject({ - synced: true, - sharedMemorySynced: false, - metaSynced: true, - }); + expect(result.job.status).not.toBe('done'); + expect(result.job.status).toBe('unreachable'); + expect(result.state).toMatchObject({ synced: false }); expect(result.readiness).toMatchObject({ - durableVerified: true, + durableVerified: false, sharedMemoryVerified: false, }); }); + it('does not settle as done when a data-bearing peer failed and an unrelated peer answered empty', async () => { + // The reported field shape: 122,705 data triples fetched, five failed + // phases, nothing verified, and unrelated peers answering empty — which + // previously settled the job as `done` with 1 KA out of 40. + const masked = cleanEmptyResult(); + masked.connectedPeers = 6; + masked.totalPeers = 6; + masked.selectedPeers = 6; + masked.syncCapablePeers = 6; + masked.peersTried = 6; + masked.peersResponded = 6; + if (!masked.diagnostics?.durable) throw new Error('durable diagnostics missing'); + masked.diagnostics.durable.fetchedDataTriples = 122_705; + masked.diagnostics.durable.failedPhases = 5; + + const result = await subscribe({ + hasConfirmedMeta: true, + includeSharedMemory: false, + result: masked, + initial: { + subscribed: true, + synced: false, + sharedMemorySynced: false, + metaSynced: true, + }, + }); + + expect(result.job.status).not.toBe('done'); + expect(result.state).toMatchObject({ synced: false }); + expect(result.readiness).toMatchObject({ durableVerified: false }); + }); + it('does not promote private data readiness from unrelated empty responders after metadata is local', async () => { const result = await subscribe({ hasConfirmedMeta: true, diff --git a/packages/cli/vitest.unit.config.ts b/packages/cli/vitest.unit.config.ts index c1dee3b3fe..04464f9d5e 100644 --- a/packages/cli/vitest.unit.config.ts +++ b/packages/cli/vitest.unit.config.ts @@ -67,6 +67,8 @@ export default defineConfig({ 'test/random-sampling-status.test.ts', 'test/catchup-runner.test.ts', 'test/catchup-runner-worker-impl.test.ts', + 'test/catchup-runner-worker-lifecycle.test.ts', + 'test/catchup-runner-worker-killswitch.test.ts', 'test/relay-status-block.test.ts', 'test/supervisor-liveness.test.ts', 'test/promote-async-routes.test.ts',