From b93a473a8f099df60682db4d063baf361ba664f6 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 02:30:51 +0200 Subject: [PATCH 01/44] fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006) Foreground Context Graph catch-up pulled the whole graph from EVERY sync-capable peer. On a 14-peer testnet that is 5-13 redundant full payloads (147,246 fetched triples for a 24,541-triple graph, ~278MB), which saturates the node-wide `sync-global` scheduler and displaces background work. Separately, a clean EMPTY response from an unrelated peer proved a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 KA out of 40. * Progressive peer walk. The peer list already arrives ranked authority-first, but that ordering never became selection. Contact peers in escalating waves (1 -> 2 -> 4, capped by the existing concurrency knob) and stop as soon as every requested plane is proven by verified data; narrow fallback peers to the planes still in question. Wave 1 is the curator when one is resolvable, so the happy path transfers exactly one payload. `DKG_CATCHUP_STOP_ON_PROOF=0` restores the previous full fan-out. * Fail-closed empty proof. An empty response cannot distinguish a peer hosting an empty graph from one that never heard of it: the requester emits `emptyResponses` only when BOTH phase payloads are empty, so an empty answer can never carry hosting evidence. Emptiness is therefore only provable as a whole-round verdict - at least one clean empty completion, nobody delivered any content, and nothing failed, timed out, was denied, or was deferred. One clean empty peer can no longer mask a data-bearing peer that failed. Empty never stops the walk, so that denominator is always complete. * Wall-clock backpressure budget. The foreground retry ladder was a fixed [100, 250, 500] - 850ms total - against admitted rounds bounded by SYNC_TOTAL_TIMEOUT_MS (120s) and measured queue waits of 87-109s, so a refused admission always exhausted its budget before the head of the queue could clear. Replaced with bounded exponential backoff plus jitter against an absolute per-plane deadline (DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, default 60s). The sleep is unref'd so a pending backoff cannot outlive agent.stop(). * Queue-origin observability. `sync-global` admissions now carry a bounded `SyncAdmissionSource`, so the operation dimension reads `durable:catchup-foreground` instead of duplicating `lane`. Existing diagnostics and `[backpressure]` log records already expose per- operation counts and queue/active ages, so this attributes pressure to a trigger with no core telemetry change and no new cardinality risk - unknown origins clamp to `unspecified`. * Worker exit safety. `close()` terminates the Worker, which emits 'exit' and never 'error', so a pending run promise was never settled and the fire-and-forget subscribe job stayed `running` forever. Deliberate tradeoff: a peer's `complete` flag proves it served its own manifest, not that the manifest was network-complete, and shared memory has no completion flag at all. Foreground catch-up is therefore optimised for one fast authoritative payload; breadth and eventual convergence remain the background reconcile lane's job, which this change leaves fanning out untouched. Co-Authored-By: Claude Opus 5 (1M context) --- docs/use-dkg/backpressure-observability.md | 26 ++ packages/agent/src/dkg-agent-lifecycle.ts | 39 ++- packages/agent/src/index.ts | 20 +- packages/agent/src/sync/backpressure.ts | 31 +- .../agent/src/sync/catchup-concurrency.ts | 38 +++ packages/agent/src/sync/catchup-policy.ts | 121 +++++++- packages/agent/src/sync/policy.ts | 39 +++ packages/agent/test/agent.part-16.test.ts | 7 +- .../agent/test/catchup-concurrency.test.ts | 46 +++ packages/agent/test/catchup-policy.test.ts | 163 ++++++++-- packages/agent/test/sync-backpressure.test.ts | 48 ++- packages/agent/test/sync-policy.test.ts | 32 ++ packages/agent/vitest.unit.config.ts | 5 + packages/cli/src/api-client.ts | 6 + .../cli/src/catchup-runner-worker-impl.ts | 264 ++++++++++------- packages/cli/src/catchup-runner.ts | 134 ++++++++- packages/cli/src/cli-helpers.ts | 4 +- packages/cli/src/context-graph-readiness.ts | 21 +- .../test/catchup-runner-worker-impl.test.ts | 278 +++++++++++++++--- .../catchup-runner-worker-lifecycle.test.ts | 38 +++ packages/cli/test/catchup-runner.test.ts | 100 +++++++ .../context-graph-catchup-readiness.test.ts | 81 ++++- .../context-graph-subscribe-readiness.test.ts | 50 +++- packages/cli/vitest.unit.config.ts | 1 + 24 files changed, 1359 insertions(+), 233 deletions(-) create mode 100644 packages/agent/test/catchup-concurrency.test.ts create mode 100644 packages/cli/test/catchup-runner-worker-lifecycle.test.ts diff --git a/docs/use-dkg/backpressure-observability.md b/docs/use-dkg/backpressure-observability.md index 6ea8aec4a6..d79ffe5f2b 100644 --- a/docs/use-dkg/backpressure-observability.md +++ b/docs/use-dkg/backpressure-observability.md @@ -69,6 +69,32 @@ The first registered sources are: `normal`, and `background` lanes; - `sync-global`: the process-wide sync admission queue and its sync lanes. +### Attributing `sync-global` pressure to a trigger + +The `lane` of a `sync-global` entry says *what kind of work* is queued +(`durable`, `changelog`, `shared_memory`, `swm_recovery`), but every trigger +funnels into the same few lanes. Its `operation` label therefore pairs the +collapsed work class with the **admission source** — the trigger that enqueued +it — as `:`: + +| Source | Trigger | +| --- | --- | +| `catchup-foreground` | explicit Context Graph catch-up (`POST /api/context-graph/subscribe`) | +| `catchup-background` | automatic post-approval / reconcile catch-up | +| `on-connect` | sync-on-connect after a peer dial | +| `reconcile` | the periodic sync reconciler | +| `vm-recovery` | foreground repair of specific missing Knowledge Assets | +| `swm-recovery` | curator-targeted shared-memory recovery | +| `unspecified` | a caller that did not declare an origin | + +So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}` +in a `queuedOperations` summary reads as "four explicit catch-up durable +admissions are queued, the oldest for 109 seconds", and the matching +`activeOperations` entry gives the same view for admitted work. Both halves are +closed sets, so the label space stays bounded (5 × 7) and, as before, no Context +Graph id or peer id ever reaches a metric, log line, or diagnostics response — +an unrecognized source is clamped to `unspecified`. + Other schedulers can extend `ObservableScheduler` and call its protected lifecycle methods at their existing admission boundaries. They keep complete ownership of policy. diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index c6ae75b068..1adcfd049b 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -315,6 +315,7 @@ import { countSyncPriorityClasses, orderContextGraphIdsByPriority, syncPriorityClass, + type SyncAdmissionSource, type SyncSchedulerLane, } from './sync/policy.js'; import { @@ -939,6 +940,12 @@ export type DurableSyncOptions = { exactAssetUals?: string[]; /** Admission override for foreground VM recovery. */ priority?: number; + /** + * Which trigger asked for this sync. Recorded as a bounded dimension on + * node-wide scheduler diagnostics so queue pressure can be attributed to an + * origin; clamped to the closed `SyncAdmissionSource` set before use. + */ + source?: string; }; type LegacyDurableContextGraphOptions = { @@ -1154,6 +1161,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { work: () => Promise, priorityOverride?: number, operationSignal?: AbortSignal, + source?: string, ): Promise { const priority = priorityOverride ?? contextGraphPriority(this.config.syncContextGraphPriorities, contextGraphId); @@ -1171,6 +1179,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { lane, priority, priorityClass: syncPriorityClass(priority), + source, signal: admissionBoundary.signal, logInfo: (opCtx, message) => this.log.info(opCtx, message), }, @@ -3616,6 +3625,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { this: DKGAgent, remotePeer: string, probe: SyncReconcilerProbe, + source: SyncAdmissionSource = 'on-connect', ): Promise { const lastOk = this.lastSuccessfulSyncAt.get(remotePeer); const lastProgress = this.lastSyncProgressAt.get(remotePeer); @@ -3623,7 +3633,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { try { const outcome = await this.trySyncFromPeer(remotePeer, () => { syncAccountingClearedBackoff = true; - }); + }, source); if (outcome === 'deferred-backpressure') { this.log.info( createOperationContext('sync'), @@ -3671,6 +3681,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { this: DKGAgent, remotePeer: string, onSyncAccounting?: (outcome: SyncOnConnectPeerOutcome) => void, + source: SyncAdmissionSource = 'on-connect', ): Promise { if (!this.started) { return 'not-started'; @@ -3708,12 +3719,13 @@ export class LifecycleSyncMethods extends DKGAgentBase { undefined, undefined, undefined, - { stopOnBackoffWorthyFailure: true }, + { stopOnBackoffWorthyFailure: true, source }, ), refreshMetaSyncedFlags: (contextGraphIds) => this.refreshMetaSyncedFlags(contextGraphIds), discoverContextGraphsFromStore: () => this.discoverContextGraphsFromStore(), syncSharedMemoryFromPeer: async (peerId, contextGraphIds) => this.syncSharedMemoryFromPeerDetailed(peerId, contextGraphIds, { stopOnBackoffWorthyFailure: true, + source, sharedMemorySyncPlan: await getSharedMemorySyncPlan(peerId), }), syncSharedMemoryOnConnect: syncOnConnectEnabled(this.config) && (this.config.syncSharedMemoryOnConnect ?? true), @@ -3991,7 +4003,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { if (!(await this.ensurePeerAdmittedForRecovery(peerId, ctx, 'Sync reconciler'))) continue; const shortPeer = peerId.slice(-8); this.log.info(ctx, `Sync reconciler retrying ${shortPeer} (last success: ${lastOk == null ? 'never' : `${Math.round((now - lastOk) / 1000)}s ago`}${backoff ? `, prior failures: ${backoff.failures}` : ''})`); - this.attemptSyncFromPeerWithReconcilerAccounting(peerId, probe) + this.attemptSyncFromPeerWithReconcilerAccounting(peerId, probe, 'reconcile') .then(() => undefined) .catch((err: unknown) => { const message = err instanceof Error ? err.message : String(err); @@ -4361,6 +4373,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds, onAccessDenied, options?.priority, + options?.source, ); changelogResult = lane.result; legacyContextGraphIds = lane.remainingLegacyCgs; @@ -4460,6 +4473,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { work, options?.priority, operationBoundary.signal, + options?.source, ), operationBoundary.signal, ); @@ -4539,6 +4553,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { exactAssetUals: assetUals, stopOnBackoffWorthyFailure: true, priority: 1_000, + source: 'vm-recovery', }, ); } @@ -4736,6 +4751,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds: string[], onAccessDenied?: (contextGraphId: string) => void, priority?: number, + source?: string, ): Promise<{ result?: DurableSyncResult; remainingLegacyCgs: string[] }> { const peerProtocols = await this.getPeerProtocols(remotePeerId); if (!peerProtocols.includes(PROTOCOL_SYNC_CHANGELOG)) { @@ -4777,6 +4793,8 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.operationId, run, priority, + undefined, + source, ), merge: mergeDurableSyncAccumulatorInto, markDeferred: (summary) => { @@ -5212,6 +5230,8 @@ export class LifecycleSyncMethods extends DKGAgentBase { sharedMemorySyncPlan?: SharedMemorySyncContextGraphPlan; /** Admission override for foreground catch-up. */ priority?: number; + /** Bounded admission origin for node-wide scheduler diagnostics. */ + source?: string; }, ): Promise { const ctx = createOperationContext('sync'); @@ -5437,6 +5457,8 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.operationId, run, options?.priority, + undefined, + options?.source, ), merge: mergeSharedMemorySyncResults, markDeferred: (summary) => ({ @@ -5520,6 +5542,9 @@ export class LifecycleSyncMethods extends DKGAgentBase { remotePeerId, contextGraphId, ), + undefined, + undefined, + 'swm-recovery', ); } @@ -5865,18 +5890,18 @@ export class LifecycleSyncMethods extends DKGAgentBase { return runCatchupPlanesWithPolicy({ mode, includeSharedMemory, - syncDurable: ({ priority }) => this.syncFromPeerDetailed( + syncDurable: ({ priority, source }) => this.syncFromPeerDetailed( remotePeerId, [contextGraphId], undefined, undefined, undefined, - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ).catch(() => createFailedPeerDurableSyncResult()), - syncSharedMemory: ({ priority }) => this.syncSharedMemoryFromPeerDetailed( + syncSharedMemory: ({ priority, source }) => this.syncSharedMemoryFromPeerDetailed( remotePeerId, [contextGraphId], - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ).catch(emptyShared), }); }, diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index cae5889e80..ccab71c045 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -127,12 +127,15 @@ export type { AcceptedRfc64CatalogAccessSnapshotV1, } from './rfc64/catalog-access-policy-v1.js'; export { + SYNC_ADMISSION_SOURCES, contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, normalizeSyncContextGraphPriorities, orderContextGraphIdsByPriority, syncPriorityClass, validateSyncResponderSnapshotLimitsConfig, + type SyncAdmissionSource, type SyncContextGraphPriorityConfig, type SyncPriorityClass, type SyncResponderSnapshotLimitsConfig, @@ -304,14 +307,27 @@ export { // registry-scale per-peer fan-out and must be bounded by the SAME knob, without // deep-importing the compiled `dist/` module. export { mapWithConcurrency } from './map-with-concurrency.js'; -export { CATCHUP_MAX_CONCURRENT_PEER_SYNCS } from './sync/catchup-concurrency.js'; export { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, + catchupWaveSizes, +} from './sync/catchup-concurrency.js'; +export { + CATCHUP_BACKPRESSURE_BASE_DELAY_MS, + CATCHUP_BACKPRESSURE_JITTER_RATIO, + CATCHUP_BACKPRESSURE_MAX_DELAY_MS, + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, FOREGROUND_CATCHUP_SYNC_PRIORITY, catchupPriorityForMode, + catchupSourceForMode, + nextCatchupBackpressureDelayMs, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, + type CatchupAdmissionSource, + type CatchupBackpressureRetryPolicy, type CatchupMode, type CatchupPlaneContext, + type CatchupPlanePolicyClock, type CatchupPlanePolicyOptions, type CatchupPlanePolicyResult, type CatchupPlaneResult, diff --git a/packages/agent/src/sync/backpressure.ts b/packages/agent/src/sync/backpressure.ts index 99ccc4b7c0..6feb97d362 100644 --- a/packages/agent/src/sync/backpressure.ts +++ b/packages/agent/src/sync/backpressure.ts @@ -1,5 +1,10 @@ import { getMetrics, type OperationContext } from '@origintrail-official/dkg-core'; -import type { SyncPriorityClass, SyncSchedulerLane } from './policy.js'; +import { + normalizeSyncAdmissionSource, + type SyncAdmissionSource, + type SyncPriorityClass, + type SyncSchedulerLane, +} from './policy.js'; import { PriorityAdmissionQueue, type PriorityAdmission, @@ -31,6 +36,7 @@ interface GlobalQueuePayload { limit: number; label: string; contextGraphId?: string; + source: SyncAdmissionSource; } export const DEFAULT_SYNC_GLOBAL_MAX_INFLIGHT = 2; @@ -50,6 +56,19 @@ function syncOperationClass(label: string): string { } } +/** + * `:` — the operation dimension of node-wide pressure + * diagnostics. The work class alone duplicates `lane`; pairing it with the + * admission source is what lets an operator attribute a saturated `sync-global` + * queue to explicit catch-up versus sync-on-connect versus reconcile, and read + * per-trigger queue/active ages straight off the snapshot. Both halves are + * closed sets (5 × 7), so the label space stays bounded and free of Context + * Graph and peer identifiers. + */ +function syncAdmissionOperation(payload: GlobalQueuePayload): string { + return `${syncOperationClass(payload.label)}:${normalizeSyncAdmissionSource(payload.source)}`; +} + let inflight = 0; let lastLimit: number | null = null; let lastQueueLimit: number | null = null; @@ -68,8 +87,9 @@ const queue = new PriorityAdmissionQueue({ observability: { scheduler: 'sync-global', // Admission labels also carry CG/peer correlation identifiers. Collapse - // them to a fixed operation class before node-wide diagnostics/logging. - operation: (entry) => syncOperationClass(entry.payload.label), + // them to a fixed operation class, paired with the bounded admission + // source, before node-wide diagnostics/logging. + operation: (entry) => syncAdmissionOperation(entry.payload), inflightLimit: (entry) => entry.payload.limit, thresholds: { degradedQueueAgeMs: DEFAULT_SYNC_PRIORITY_AGING_MS / 2, @@ -113,6 +133,7 @@ function acquire( lane: SyncSchedulerLane; priority: number; priorityClass: SyncPriorityClass; + source: SyncAdmissionSource; signal?: AbortSignal; agingThresholdMs: number; now: () => number; @@ -129,6 +150,7 @@ function acquire( limit, label: options.label, contextGraphId: options.contextGraphId, + source: options.source, }, ownerKey: 'global', lane: options.lane, @@ -251,6 +273,8 @@ export async function withGlobalSyncBackpressure( lane?: SyncSchedulerLane; priority?: number; priorityClass?: SyncPriorityClass; + /** Which trigger enqueued this admission; clamped to the closed set. */ + source?: string; signal?: AbortSignal; /** Deterministic scheduler injection; not operator configuration. */ agingThresholdMs?: number; @@ -281,6 +305,7 @@ export async function withGlobalSyncBackpressure( lane, priority, priorityClass, + source: normalizeSyncAdmissionSource(options.source), signal: options.signal, agingThresholdMs: options.agingThresholdMs ?? DEFAULT_SYNC_PRIORITY_AGING_MS, now: options.now ?? Date.now, diff --git a/packages/agent/src/sync/catchup-concurrency.ts b/packages/agent/src/sync/catchup-concurrency.ts index 83c31c0ba6..8bb916bcea 100644 --- a/packages/agent/src/sync/catchup-concurrency.ts +++ b/packages/agent/src/sync/catchup-concurrency.ts @@ -3,3 +3,41 @@ export const CATCHUP_MAX_CONCURRENT_PEER_SYNCS: number = (() => { const raw = Number(process.env.DKG_CATCHUP_MAX_CONCURRENT_PEERS); return Number.isInteger(raw) && raw > 0 ? raw : 4; })(); + +/** + * Operator kill-switch for the progressive catch-up walk (issue #2006). + * + * With it off, foreground catch-up reverts to contacting every sync-capable + * peer in one bounded pass — the pre-fix behaviour, kept reachable because the + * walk trades breadth for cost: a peer's `complete` flag only proves it served + * its own manifest, so stopping early can land one peer's snapshot instead of + * the union of every peer's. + */ +export const CATCHUP_STOP_ON_PROOF: boolean = (() => { + const raw = process.env.DKG_CATCHUP_STOP_ON_PROOF?.trim().toLowerCase(); + return !(raw === '0' || raw === 'false' || raw === 'no' || raw === 'off'); +})(); + +/** + * Escalating wave sizes for the progressive peer walk: 1, 2, 4, … capped by + * `maxConcurrency` and truncated to `peerCount`. + * + * The first wave is a single peer because the peer list is already ranked + * authority-first (preferred/curator, then known cores), so the happy path + * downloads exactly one payload. Doubling afterwards keeps the fallback tail + * short — a flat wave of `maxConcurrency` would pull that many concurrent full + * payloads before any of them could prove the plane. + */ +export function catchupWaveSizes(peerCount: number, maxConcurrency: number): number[] { + const cap = Number.isInteger(maxConcurrency) && maxConcurrency > 0 ? maxConcurrency : 1; + const sizes: number[] = []; + let remaining = Math.max(0, Math.trunc(peerCount)); + let size = 1; + while (remaining > 0) { + const take = Math.min(size, remaining); + sizes.push(take); + remaining -= take; + size = Math.min(cap, size * 2); + } + return sizes; +} diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 97d001dd90..920e4b8c16 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -1,7 +1,33 @@ export type CatchupMode = 'background' | 'foreground'; export const FOREGROUND_CATCHUP_SYNC_PRIORITY = 2_000; -export const CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS = [100, 250, 500] as const; + +/** First backoff step after a foreground plane is refused by local admission. */ +export const CATCHUP_BACKPRESSURE_BASE_DELAY_MS = 250; +/** Ceiling for one backoff step; the scheduler drains in seconds, not minutes. */ +export const CATCHUP_BACKPRESSURE_MAX_DELAY_MS = 5_000; +/** Fraction of a delay that jitter may add, so parallel receivers desynchronize. */ +export const CATCHUP_BACKPRESSURE_JITTER_RATIO = 0.25; + +/** + * How long one foreground plane may keep waiting for local scheduler capacity. + * + * The previous policy was a fixed `[100, 250, 500]` ladder — 850 ms in total — + * while an admitted `sync-global` round is bounded by `SYNC_TOTAL_TIMEOUT_MS` + * (120 s) per plane, and issue #2006 measured queue waits of 87–109 s. A refused + * foreground admission therefore always exhausted its budget long before the + * head of the queue could possibly have cleared. Waiting costs a timer and no + * work, so the budget is now wall-clock and generous — but still bounded, so a + * permanently saturated node fails the catch-up job instead of pinning it at + * `running` forever. + */ +export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = (() => { + const raw = Number(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + return Number.isInteger(raw) && raw >= 0 ? raw : 60_000; +})(); + +/** Bounded admission origin recorded on node-wide scheduler diagnostics. */ +export type CatchupAdmissionSource = 'catchup-foreground' | 'catchup-background'; export interface CatchupPlaneResult { deferredBackpressure?: number; @@ -9,18 +35,33 @@ export interface CatchupPlaneResult { export interface CatchupPlaneContext { priority?: number; + source?: CatchupAdmissionSource; +} + +export interface CatchupBackpressureRetryPolicy { + baseDelayMs?: number; + maxDelayMs?: number; + jitterRatio?: number; + /** Total wall-clock budget for one plane's admission retries. */ + maxWaitMs?: number; +} + +/** Deterministic seams for tests; never operator configuration. */ +export interface CatchupPlanePolicyClock { + retry?: CatchupBackpressureRetryPolicy; + wait?: (delayMs: number) => Promise; + now?: () => number; + random?: () => number; } export interface CatchupPlanePolicyOptions< TDurable extends CatchupPlaneResult, TShared extends CatchupPlaneResult, -> { +> extends CatchupPlanePolicyClock { mode: CatchupMode; includeSharedMemory: boolean; syncDurable: (context: CatchupPlaneContext) => Promise; syncSharedMemory: (context: CatchupPlaneContext) => Promise; - retryDelaysMs?: readonly number[]; - wait?: (delayMs: number) => Promise; } export interface CatchupPlanePolicyResult< @@ -35,25 +76,75 @@ export function catchupPriorityForMode(mode: CatchupMode): number | undefined { return mode === 'foreground' ? FOREGROUND_CATCHUP_SYNC_PRIORITY : undefined; } -async function runCatchupPlane( +export function catchupSourceForMode(mode: CatchupMode): CatchupAdmissionSource { + return mode === 'foreground' ? 'catchup-foreground' : 'catchup-background'; +} + +/** A pending backoff must never keep the process alive past `agent.stop()`. */ +function defaultWait(delayMs: number): Promise { + return new Promise((resolve) => { + const timer = setTimeout(resolve, delayMs); + if (typeof timer.unref === 'function') timer.unref(); + }); +} + +/** + * Exponential backoff with additive jitter, clamped so a sleep never runs past + * the plane's retry deadline. Returns `undefined` once no useful wait remains. + */ +export function nextCatchupBackpressureDelayMs(input: { + attempt: number; + remainingMs: number; + policy?: CatchupBackpressureRetryPolicy; + random?: () => number; +}): number | undefined { + if (input.remainingMs <= 0) return undefined; + const baseDelayMs = input.policy?.baseDelayMs ?? CATCHUP_BACKPRESSURE_BASE_DELAY_MS; + const maxDelayMs = input.policy?.maxDelayMs ?? CATCHUP_BACKPRESSURE_MAX_DELAY_MS; + const jitterRatio = input.policy?.jitterRatio ?? CATCHUP_BACKPRESSURE_JITTER_RATIO; + const exponential = Math.min(maxDelayMs, baseDelayMs * 2 ** Math.max(0, input.attempt)); + const jittered = exponential * (1 + jitterRatio * (input.random?.() ?? Math.random())); + return Math.max(1, Math.min(Math.round(jittered), input.remainingMs)); +} + +/** + * Run one catch-up plane, retrying only while LOCAL admission backpressure kept + * refusing it, until a bounded wall-clock deadline. + * + * Cancellation needs no extra plumbing: an aborted admission raises an + * `AbortError`, not a `SyncBackpressureBusyError`, so it never sets + * `deferredBackpressure` and the loop's own guard exits on the next iteration. + */ +export async function runCatchupPlaneWithPolicy( mode: CatchupMode, run: (context: CatchupPlaneContext) => Promise, - options: Pick, 'retryDelaysMs' | 'wait'>, + options: CatchupPlanePolicyClock = {}, ): Promise { - const context = { priority: catchupPriorityForMode(mode) }; + const context: CatchupPlaneContext = { + priority: catchupPriorityForMode(mode), + source: catchupSourceForMode(mode), + }; let result = await run(context); if (mode !== 'foreground') return result; - const retryDelaysMs = options.retryDelaysMs ?? CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS; - const wait = options.wait ?? ((delayMs: number) => new Promise((resolve) => { - setTimeout(resolve, delayMs); - })); - for (const delayMs of retryDelaysMs) { + const now = options.now ?? Date.now; + const wait = options.wait ?? defaultWait; + const maxWaitMs = options.retry?.maxWaitMs ?? CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + // Absolute deadline fixed once per plane, so retries cannot compound with the + // time the refused rounds themselves consumed. + const retryUntil = now() + maxWaitMs; + for (let attempt = 0; ; attempt += 1) { if ((result.deferredBackpressure ?? 0) === 0) return result; + const delayMs = nextCatchupBackpressureDelayMs({ + attempt, + remainingMs: retryUntil - now(), + policy: options.retry, + random: options.random, + }); + if (delayMs === undefined) return result; await wait(delayMs); result = await run(context); } - return result; } /** @@ -68,11 +159,11 @@ export async function runCatchupPlanesWithPolicy< >( options: CatchupPlanePolicyOptions, ): Promise> { - const durable = await runCatchupPlane(options.mode, options.syncDurable, options); + const durable = await runCatchupPlaneWithPolicy(options.mode, options.syncDurable, options); if (!options.includeSharedMemory || (durable.deferredBackpressure ?? 0) > 0) { return { durable, shared: null }; } - const shared = await runCatchupPlane(options.mode, options.syncSharedMemory, options); + const shared = await runCatchupPlaneWithPolicy(options.mode, options.syncSharedMemory, options); return { durable, shared }; } diff --git a/packages/agent/src/sync/policy.ts b/packages/agent/src/sync/policy.ts index 975147370d..acd748c3ed 100644 --- a/packages/agent/src/sync/policy.ts +++ b/packages/agent/src/sync/policy.ts @@ -24,6 +24,45 @@ export type SyncSchedulerLane = | 'pre_authorization' | 'responder'; +/** + * Which trigger enqueued a `sync-global` admission. + * + * The lane says WHAT kind of work is queued; every trigger funnels into the same + * few lanes, so lane alone cannot tell an operator whether a saturated queue is + * an explicit user-driven catch-up, routine sync-on-connect, or a background + * reconcile. Issue #2006 had to reconstruct that from daemon logs. + * + * The set is deliberately closed and small: these values become metric and log + * dimensions, so cardinality is a contract, not an implementation detail. + */ +export const SYNC_ADMISSION_SOURCES = [ + 'catchup-foreground', + 'catchup-background', + 'on-connect', + 'reconcile', + 'vm-recovery', + 'swm-recovery', + 'unspecified', +] as const; + +export type SyncAdmissionSource = typeof SYNC_ADMISSION_SOURCES[number]; + +const SYNC_ADMISSION_SOURCE_SET: ReadonlySet = new Set(SYNC_ADMISSION_SOURCES); + +/** + * Clamp an admission origin to the closed set before it becomes a diagnostic + * label. The union is compile-time only; a value crossing a worker/RPC boundary + * or arriving through a cast must never be able to widen the label space or + * smuggle a Context Graph / peer identifier into node-wide diagnostics. + */ +export function normalizeSyncAdmissionSource( + source: string | undefined, +): SyncAdmissionSource { + return source !== undefined && SYNC_ADMISSION_SOURCE_SET.has(source) + ? source as SyncAdmissionSource + : 'unspecified'; +} + const SNAPSHOT_LIMIT_PATHS = [ ['global', 'rows'], ['global', 'bytesEstimate'], diff --git a/packages/agent/test/agent.part-16.test.ts b/packages/agent/test/agent.part-16.test.ts index 2587ee8993..cceee3a2e2 100644 --- a/packages/agent/test/agent.part-16.test.ts +++ b/packages/agent/test/agent.part-16.test.ts @@ -165,18 +165,21 @@ describe('DKGAgent config — syncContextGraphs and queryAccess warning', () => }); expect(peerStoreReads).toBe(3); + // Background catch-up carries no admission priority override, but it + // does tag its origin so node-wide scheduler diagnostics can attribute + // queue pressure to a trigger (issue #2006). expect(syncFromPeerDetailed.calls.at(-1)).toEqual([ remotePeer.toString(), ['runtime-contextGraph'], undefined, undefined, undefined, - undefined, + { source: 'catchup-background' }, ]); expect(syncSharedMemoryFromPeerDetailed.calls.at(-1)).toEqual([ remotePeer.toString(), ['runtime-contextGraph'], - undefined, + { source: 'catchup-background' }, ]); expect(result.connectedPeers).toBe(1); expect(result.syncCapablePeers).toBe(1); diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts new file mode 100644 index 0000000000..d2a43a4144 --- /dev/null +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -0,0 +1,46 @@ +import { describe, expect, it } from 'vitest'; +import { + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + catchupWaveSizes, +} from '../src/sync/catchup-concurrency.js'; + +describe('catchupWaveSizes', () => { + it('starts with a single peer so a proving authority costs one payload', () => { + // The peer list arrives ranked authority-first, so wave 1 is the curator + // whenever one is resolvable. Issue #2006: the pre-fix fan-out pulled the + // whole graph from every sync-capable peer instead. + expect(catchupWaveSizes(14, 4)[0]).toBe(1); + expect(catchupWaveSizes(1, 4)).toEqual([1]); + }); + + it('escalates by doubling up to the concurrency cap', () => { + expect(catchupWaveSizes(14, 4)).toEqual([1, 2, 4, 4, 3]); + expect(catchupWaveSizes(20, 4)).toEqual([1, 2, 4, 4, 4, 4, 1]); + expect(catchupWaveSizes(7, 8)).toEqual([1, 2, 4]); + }); + + it('never exceeds the cap or the peer count', () => { + for (const cap of [1, 2, 3, 4, 8]) { + for (const peerCount of [0, 1, 3, 5, 13, 40]) { + const sizes = catchupWaveSizes(peerCount, cap); + expect(sizes.reduce((sum, size) => sum + size, 0)).toBe(peerCount); + for (const size of sizes) { + expect(size).toBeGreaterThan(0); + expect(size).toBeLessThanOrEqual(cap); + } + } + } + }); + + it('degrades to serial waves for a non-positive cap instead of looping forever', () => { + expect(catchupWaveSizes(3, 0)).toEqual([1, 1, 1]); + expect(catchupWaveSizes(3, Number.NaN)).toEqual([1, 1, 1]); + expect(catchupWaveSizes(0, 4)).toEqual([]); + expect(catchupWaveSizes(-2, 4)).toEqual([]); + }); + + it('keeps the shared fan-out cap a small positive number', () => { + expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); + expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeLessThanOrEqual(16); + }); +}); diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index 59ca4505e1..f4eeb84925 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -1,22 +1,47 @@ import { describe, expect, it, vi } from 'vitest'; import { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, + CATCHUP_BACKPRESSURE_BASE_DELAY_MS, + CATCHUP_BACKPRESSURE_MAX_DELAY_MS, FOREGROUND_CATCHUP_SYNC_PRIORITY, + nextCatchupBackpressureDelayMs, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, } from '../src/sync/catchup-policy.js'; +/** + * A virtual clock whose `wait` advances the clock by exactly the requested + * delay. This makes the wall-clock retry budget deterministic: the number of + * attempts is a pure function of the injected budget and the backoff curve. + */ +function virtualClock(startMs = 1_000) { + let nowMs = startMs; + const waits: number[] = []; + return { + waits, + now: () => nowMs, + wait: async (delayMs: number) => { + waits.push(delayMs); + nowMs += delayMs; + }, + elapsed: () => nowMs - startMs, + }; +} + describe('runCatchupPlanesWithPolicy', () => { - it('derives foreground priority and retries durable before starting SWM', async () => { + it('derives foreground priority and source and retries durable before starting SWM', async () => { const order: string[] = []; const priorities: Array = []; - const waits: number[] = []; - const syncDurable = vi.fn(async ({ priority }: { priority?: number }) => { + const sources: Array = []; + const clock = virtualClock(); + const syncDurable = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); order.push(`durable-${syncDurable.mock.calls.length}`); return { deferredBackpressure: syncDurable.mock.calls.length === 1 ? 1 : 0 }; }); - const syncSharedMemory = vi.fn(async ({ priority }: { priority?: number }) => { + const syncSharedMemory = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); order.push('shared'); return { deferredBackpressure: 0 }; }); @@ -26,8 +51,9 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - retryDelaysMs: [3, 5], - wait: async (delayMs) => { waits.push(delayMs); }, + now: clock.now, + wait: clock.wait, + random: () => 0, }); expect(result).toEqual({ @@ -40,10 +66,16 @@ describe('runCatchupPlanesWithPolicy', () => { FOREGROUND_CATCHUP_SYNC_PRIORITY, FOREGROUND_CATCHUP_SYNC_PRIORITY, ]); - expect(waits).toEqual([3]); + expect(sources).toEqual([ + 'catchup-foreground', + 'catchup-foreground', + 'catchup-foreground', + ]); + expect(clock.waits).toEqual([CATCHUP_BACKPRESSURE_BASE_DELAY_MS]); }); it('retries only SWM when durable already completed', async () => { + const clock = virtualClock(); const syncDurable = vi.fn(async () => ({ deferredBackpressure: 0 })); const syncSharedMemory = vi.fn() .mockResolvedValueOnce({ deferredBackpressure: 1 }) @@ -54,8 +86,8 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - retryDelaysMs: [1], - wait: async () => {}, + now: clock.now, + wait: clock.wait, }); expect(result.shared?.deferredBackpressure).toBe(0); @@ -63,7 +95,12 @@ describe('runCatchupPlanesWithPolicy', () => { expect(syncSharedMemory).toHaveBeenCalledTimes(2); }); - it('returns the final durable deferral without starting dependent SWM', async () => { + it('retries a deferred plane on a wall-clock budget, not a fixed attempt count', async () => { + // Before #2006 the budget was the fixed ladder [100, 250, 500] — exactly + // four attempts totalling 850 ms — against measured sync-global queue waits + // of 87-109 s, so a refused foreground admission could never outlast the + // head of the queue. The budget is now wall-clock. + const clock = virtualClock(); const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); const syncSharedMemory = vi.fn(async () => ({ deferredBackpressure: 0 })); @@ -72,21 +109,81 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, - wait: async () => {}, + retry: { maxWaitMs: 90_000 }, + now: clock.now, + wait: clock.wait, + random: () => 0, }); expect(result.durable.deferredBackpressure).toBe(1); expect(result.shared).toBeNull(); - expect(syncDurable).toHaveBeenCalledTimes( - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS.length + 1, - ); expect(syncSharedMemory).not.toHaveBeenCalled(); + // Far past the four attempts the fixed ladder allowed. + expect(syncDurable.mock.calls.length).toBeGreaterThan(4); + // …and bounded by the budget rather than running forever. + expect(clock.elapsed()).toBeLessThanOrEqual(90_000); + expect(clock.waits.reduce((sum, value) => sum + value, 0)).toBe(clock.elapsed()); + }); + + it('never sleeps past the retry deadline', async () => { + const clock = virtualClock(); + const deadlineMs = 1_000; + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + const startedAt = clock.now(); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: deadlineMs }, + now: clock.now, + wait: clock.wait, + random: () => 1, + }); + + expect(clock.elapsed()).toBeLessThanOrEqual(deadlineMs); + expect(clock.now() - startedAt).toBeLessThanOrEqual(deadlineMs); + expect(syncDurable.mock.calls.length).toBeGreaterThan(1); + }); + + it('gives up immediately when no retry budget remains', async () => { + const clock = virtualClock(); + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + + const result = await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 0 }, + now: clock.now, + wait: clock.wait, + }); + + expect(result.deferredBackpressure).toBe(1); + expect(syncDurable).toHaveBeenCalledTimes(1); + expect(clock.waits).toEqual([]); }); - it('keeps background catch-up best-effort without retries or priority', async () => { + it('retries only planes deferred by local admission backpressure', async () => { + // A timeout or transport failure is not scheduler pressure: retrying it + // here would multiply exactly the traffic issue #2006 is about. + const clock = virtualClock(); + const syncDurable = vi.fn(async () => ({ + deferredBackpressure: 0, + timedOutPhases: 1, + failedPeers: 1, + })); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 600_000 }, + now: clock.now, + wait: clock.wait, + }); + + expect(syncDurable).toHaveBeenCalledTimes(1); + expect(clock.waits).toEqual([]); + }); + + it('keeps background catch-up best-effort without retries, priority, or a foreground source', async () => { const priorities: Array = []; - const syncDurable = vi.fn(async ({ priority }: { priority?: number }) => { + const sources: Array = []; + const syncDurable = vi.fn(async ({ priority, source }: { priority?: number; source?: string }) => { priorities.push(priority); + sources.push(source); return { deferredBackpressure: 1 }; }); const syncSharedMemory = vi.fn(async () => ({ deferredBackpressure: 0 })); @@ -96,6 +193,7 @@ describe('runCatchupPlanesWithPolicy', () => { includeSharedMemory: true, syncDurable, syncSharedMemory, + retry: { maxWaitMs: 600_000 }, wait: async () => { throw new Error('background mode must not wait'); }, }); @@ -104,5 +202,36 @@ describe('runCatchupPlanesWithPolicy', () => { expect(syncDurable).toHaveBeenCalledTimes(1); expect(syncSharedMemory).not.toHaveBeenCalled(); expect(priorities).toEqual([undefined]); + expect(sources).toEqual(['catchup-background']); + }); +}); + +describe('nextCatchupBackpressureDelayMs', () => { + it('grows exponentially and clamps at the per-step ceiling', () => { + const delays = Array.from({ length: 8 }, (_, attempt) => nextCatchupBackpressureDelayMs({ + attempt, + remainingMs: Number.MAX_SAFE_INTEGER, + random: () => 0, + })); + + expect(delays[0]).toBe(CATCHUP_BACKPRESSURE_BASE_DELAY_MS); + for (let i = 1; i < delays.length; i += 1) { + expect(delays[i]!).toBeGreaterThanOrEqual(delays[i - 1]!); + } + expect(delays.at(-1)).toBe(CATCHUP_BACKPRESSURE_MAX_DELAY_MS); + }); + + it('applies jitter so parallel receivers do not retry in lockstep', () => { + const low = nextCatchupBackpressureDelayMs({ attempt: 3, remainingMs: 1e9, random: () => 0 }); + const high = nextCatchupBackpressureDelayMs({ attempt: 3, remainingMs: 1e9, random: () => 1 }); + + expect(low).toBeLessThan(high!); + expect(low).toBeGreaterThanOrEqual(CATCHUP_BACKPRESSURE_BASE_DELAY_MS); + }); + + it('never returns a delay that overshoots the remaining budget', () => { + expect(nextCatchupBackpressureDelayMs({ attempt: 10, remainingMs: 40, random: () => 1 })).toBe(40); + expect(nextCatchupBackpressureDelayMs({ attempt: 0, remainingMs: 0 })).toBeUndefined(); + expect(nextCatchupBackpressureDelayMs({ attempt: 0, remainingMs: -5 })).toBeUndefined(); }); }); diff --git a/packages/agent/test/sync-backpressure.test.ts b/packages/agent/test/sync-backpressure.test.ts index 9c43aeaaa1..f3e97b31e3 100644 --- a/packages/agent/test/sync-backpressure.test.ts +++ b/packages/agent/test/sync-backpressure.test.ts @@ -228,6 +228,7 @@ describe('sync global backpressure', () => { policy, ctx, label: 'durable:urn:cg:private:peer-a', + source: 'catchup-foreground', }, async () => new Promise((resolve) => { releaseRunning = resolve; @@ -239,6 +240,7 @@ describe('sync global backpressure', () => { policy, ctx, label: 'swm-recovery:urn:cg:private:peer-b', + source: 'reconcile', }, async () => undefined, ); @@ -247,10 +249,14 @@ describe('sync global backpressure', () => { const snapshot = backpressureRegistry.capture().schedulers.find( (scheduler) => scheduler.scheduler === 'sync-global', ); + // The operation dimension pairs the collapsed work class with the bounded + // admission origin, so a saturated queue can be attributed to a trigger + // (issue #2006 had to reconstruct that from daemon logs) without any + // Context Graph or peer identifier reaching node-wide diagnostics. expect(snapshot).toMatchObject({ lanes: [expect.objectContaining({ - activeOperations: [expect.objectContaining({ operation: 'durable' })], - queuedOperations: [expect.objectContaining({ operation: 'swm-recovery' })], + activeOperations: [expect.objectContaining({ operation: 'durable:catchup-foreground' })], + queuedOperations: [expect.objectContaining({ operation: 'swm-recovery:reconcile' })], })], }); expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); @@ -261,6 +267,44 @@ describe('sync global backpressure', () => { await Promise.all([running, queued]); }); + it('clamps an unknown admission origin instead of widening the diagnostic label space', async () => { + const ctx = createOperationContext('sync'); + const policy = resolveSyncGlobalBackpressure({ + syncGlobalMaxInflight: 1, + syncGlobalQueueLimit: 1, + }); + let releaseRunning!: () => void; + // The union is compile-time only; a value crossing the worker RPC boundary + // (or an outright cast) must not be able to smuggle an identifier into a + // metric/log dimension or blow up its cardinality. + const running = withGlobalSyncBackpressure( + { + policy, + ctx, + label: 'durable:cg-x:peer-x', + source: 'leak-urn:cg:private:xyz', + }, + async () => new Promise((resolve) => { + releaseRunning = resolve; + }), + ); + await tick(); + + const snapshot = backpressureRegistry.capture().schedulers.find( + (scheduler) => scheduler.scheduler === 'sync-global', + ); + expect(snapshot).toMatchObject({ + lanes: [expect.objectContaining({ + activeOperations: [expect.objectContaining({ operation: 'durable:unspecified' })], + })], + }); + expect(JSON.stringify(snapshot)).not.toContain('leak-'); + expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); + + releaseRunning(); + await running; + }); + it('starts a later elevated CG before an earlier deprioritized queued CG', async () => { const ctx = createOperationContext('sync'); const policy = resolveSyncGlobalBackpressure({ syncGlobalMaxInflight: 1, syncGlobalQueueLimit: 3 }); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index b43efe5065..04e868e40f 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -1,7 +1,9 @@ import { describe, expect, it } from 'vitest'; import { + SYNC_ADMISSION_SOURCES, contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, normalizeSyncContextGraphPriorities, orderContextGraphIdsByPriority, syncPriorityClass, @@ -56,3 +58,33 @@ describe('sync responder snapshot config validation', () => { .toThrow(`syncResponderSnapshotLimits.${path}`); }); }); + +describe('normalizeSyncAdmissionSource', () => { + it('passes through every declared admission origin', () => { + for (const source of SYNC_ADMISSION_SOURCES) { + expect(normalizeSyncAdmissionSource(source)).toBe(source); + } + }); + + it('clamps unknown, absent, and identifier-bearing origins to `unspecified`', () => { + // These values become metric and log dimensions on the node-wide + // `sync-global` scheduler, so the label space is a contract: an unbounded + // or identifier-bearing origin would re-open the correlation-identifier + // leak that collapsing the operation label was added to close, and would + // multiply the diagnostic cardinality. + expect(normalizeSyncAdmissionSource(undefined)).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('Catchup-Foreground')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('durable:urn:cg:private:abc')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('__proto__')).toBe('unspecified'); + expect(normalizeSyncAdmissionSource('toString')).toBe('unspecified'); + }); + + it('keeps the declared origin set small and free of punctuation', () => { + expect(new Set(SYNC_ADMISSION_SOURCES).size).toBe(SYNC_ADMISSION_SOURCES.length); + expect(SYNC_ADMISSION_SOURCES.length).toBeLessThanOrEqual(12); + for (const source of SYNC_ADMISSION_SOURCES) { + expect(source).toMatch(/^[a-z][a-z-]*$/); + } + }); +}); diff --git a/packages/agent/vitest.unit.config.ts b/packages/agent/vitest.unit.config.ts index f2842594bb..51776d5c88 100644 --- a/packages/agent/vitest.unit.config.ts +++ b/packages/agent/vitest.unit.config.ts @@ -79,6 +79,11 @@ export default defineConfig({ "test/sync-fetch-coalescing.test.ts", "test/sync-fetch-coalescing-durable.test.ts", "test/sync-backpressure.test.ts", + "test/sync-policy.test.ts", + "test/catchup-policy.test.ts", + "test/catchup-concurrency.test.ts", + "test/map-with-concurrency.test.ts", + "test/peer-selection.test.ts", "test/sync-requester-priority.test.ts", "test/sync-requester-progress.test.ts", "test/rootless-durable-bounded-progress.test.ts", diff --git a/packages/cli/src/api-client.ts b/packages/cli/src/api-client.ts index 6199a6eb28..c06e4a3e48 100644 --- a/packages/cli/src/api-client.ts +++ b/packages/cli/src/api-client.ts @@ -1500,6 +1500,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; @@ -1573,6 +1575,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; @@ -1642,6 +1646,8 @@ export class ApiClient { peersTried: number; peersResponded: number; peersSucceeded: number; + /** Sync-capable peers skipped because an earlier wave already proved every requested plane. */ + peersNotAttempted?: number; deferredBackpressure: number; dataSynced: number; sharedMemorySynced: number; diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 988e151493..c064794b02 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -1,14 +1,18 @@ import { parentPort } from 'node:worker_threads'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, + catchupWaveSizes, createFailedPeerDurableSyncResult, mapWithConcurrency, + runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, } from '@origintrail-official/dkg-agent'; import { catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByData, type CatchupJobResult, type CatchupRunRequest, } from './catchup-runner.js'; @@ -50,6 +54,33 @@ parentPort!.on('message', async (message: any) => { } }); +/** A per-peer sync round; `durable` is absent when the walk skipped that plane. */ +interface PeerRound { + durable: any | null; + shared: any | null; +} + +function emptyShared() { + return { + insertedTriples: 0, + fetchedMetaTriples: 0, + fetchedDataTriples: 0, + insertedMetaTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 0, + checkpointAdvances: 0, + emptyResponses: 0, + droppedDataTriples: 0, + failedPeers: 1, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + async function runCatchup(request: CatchupRunRequest): Promise { const prepared = await invoke<{ preferredPeerId?: string; @@ -114,20 +145,11 @@ async function runCatchup(request: CatchupRunRequest): Promise }, }; - // Run per-peer syncs in parallel, but BOUNDED. The sequential version here - // used to walk the peer set one at a time, which meant a curated-CG denial - // from a 10-peer pool took 10 × (syncDurable timeout + syncSharedMemory - // timeout) to report back — often minutes. Codex N18 then parallelised this - // Worker path (the daemon `/api/context-graph/subscribe` route) with an - // unbounded `Promise.all` — which made it the 2026-07-07 mainnet sync-storm - // engine: one subscribe on a high-degree node fired a full durable+SWM pull - // at EVERY sync-capable peer at once, saturating the triple store. Mirror - // the agent-side `syncContextGraphFromConnectedPeers` fix: run the fan-out - // through `mapWithConcurrency` under the shared cap - // (CATCHUP_MAX_CONCURRENT_PEER_SYNCS, env DKG_CATCHUP_MAX_CONCURRENT_PEERS) - // so both runners have the same latency AND the same load ceiling. The - // protocol probe below is lighter but fans out over the full post-prime-dial - // peer list, so it gets the same bound. + // Probe every connected peer for PROTOCOL_SYNC up front, bounded by the shared + // catch-up cap. This stays eager on purpose: `syncCapablePeers` and + // `noProtocolPeers` are read by daemon status mapping as counts over the whole + // connected set ("no sync-capable peers found — the curator may be offline"), + // and the probe is a peerStore lookup, not a transfer. const checked = await mapWithConcurrency( prepared.peerIds, CATCHUP_MAX_CONCURRENT_PEER_SYNCS, @@ -145,96 +167,88 @@ async function runCatchup(request: CatchupRunRequest): Promise syncCapable.push(peerId); } syncCapablePeers = syncCapable.length; - peersTried = syncCapable.length; - // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we can - // from the other peers instead of failing the entire subscribe/catch-up immediately. - const emptyShared = () => ({ - insertedTriples: 0, - fetchedMetaTriples: 0, - fetchedDataTriples: 0, - insertedMetaTriples: 0, - insertedDataTriples: 0, - bytesReceived: 0, - resumedPhases: 0, - timedOutPhases: 0, - completedPhases: 0, - checkpointAdvances: 0, - emptyResponses: 0, - droppedDataTriples: 0, - failedPeers: 1, - failedPhases: 0, - deniedPhases: 0, - deferredBackpressure: 0, - }); - // Bounded fan-out (sync-storm mitigation C-1): at most - // CATCHUP_MAX_CONCURRENT_PEER_SYNCS full per-peer sync rounds in flight. - // Every sync-capable peer is still synced and the result array is unchanged - // (input order, one entry per peer, per-peer failures isolated by the - // `.catch`es inside the callback) — the load is just staggered into waves. - const perPeerResults = await mapWithConcurrency( - syncCapable, - CATCHUP_MAX_CONCURRENT_PEER_SYNCS, - async (peerId) => { - return runCatchupPlanesWithPolicy({ - mode: 'foreground', - includeSharedMemory: request.includeSharedMemory, - syncDurable: async ({ priority }) => { - const rawDurable = await invoke( - 'syncDurable', - peerId, - request.contextGraphId, - priority, - ).catch(() => createFailedPeerDurableSyncResult()); - return { - ...rawDurable, - verifiedPrivateOnlyResponses: rawDurable.verifiedPrivateOnlyResponses ?? 0, - }; - }, - syncSharedMemory: ({ priority }) => invoke( - 'syncSharedMemory', - peerId, - request.contextGraphId, - priority, - ).catch(() => emptyShared()), - }); - }, - ); - for (const { durable, shared } of perPeerResults) { + const durableProven = (): boolean => catchupPlaneProvenByData(cleanPlaneCompletions.durable); + const sharedMemoryProven = (): boolean => + catchupPlaneProvenByData(cleanPlaneCompletions.sharedMemory); + + // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we + // can from the other peers instead of failing the entire subscribe/catch-up. + const syncPeer = async (peerId: string): Promise => { + const syncDurable = ({ priority, source }: { priority?: number; source?: string }) => + invoke('syncDurable', peerId, request.contextGraphId, priority, source) + .catch(() => createFailedPeerDurableSyncResult()) + .then((rawDurable: any) => ({ + ...rawDurable, + verifiedPrivateOnlyResponses: rawDurable.verifiedPrivateOnlyResponses ?? 0, + })); + const syncSharedMemory = ({ priority, source }: { priority?: number; source?: string }) => + invoke('syncSharedMemory', peerId, request.contextGraphId, priority, source) + .catch(() => emptyShared()); + + // Narrow each fallback peer to the planes still in question. The walk only + // continues while some requested plane is unproven, and one plane is often + // proven long before the other — a Context Graph whose public VM data is + // empty can never prove its durable plane by data, so without this a single + // unproven plane would drag a full re-pull of the ALREADY PROVEN plane out + // of every remaining peer, which is the amplification this fix exists to + // remove. + // The kill-switch restores the previous fan-out faithfully: every peer, both + // requested planes, no early stop. + const needDurable = !CATCHUP_STOP_ON_PROOF || !durableProven(); + const needSharedMemory = request.includeSharedMemory + && (!CATCHUP_STOP_ON_PROOF || !sharedMemoryProven()); + if (!needDurable) { + const shared = needSharedMemory + ? await runCatchupPlaneWithPolicy('foreground', syncSharedMemory) + : null; + return { durable: null, shared }; + } + return runCatchupPlanesWithPolicy({ + mode: 'foreground', + includeSharedMemory: needSharedMemory, + syncDurable, + syncSharedMemory, + }); + }; + + const accumulate = ({ durable, shared }: PeerRound): void => { let peerDenied = false; - dataSynced += durable.insertedDataTriples ?? 0; - diagnostics.durable.fetchedMetaTriples += durable.fetchedMetaTriples; - diagnostics.durable.fetchedDataTriples += durable.fetchedDataTriples; - diagnostics.durable.insertedMetaTriples += durable.insertedMetaTriples; - diagnostics.durable.insertedDataTriples += durable.insertedDataTriples; - diagnostics.durable.bytesReceived += durable.bytesReceived; - diagnostics.durable.resumedPhases += durable.resumedPhases; - diagnostics.durable.timedOutPhases += durable.timedOutPhases ?? 0; - diagnostics.durable.completedPhases += durable.completedPhases ?? 0; - diagnostics.durable.checkpointAdvances += durable.checkpointAdvances ?? 0; - diagnostics.durable.emptyResponses += durable.emptyResponses; - diagnostics.durable.metaOnlyResponses += durable.metaOnlyResponses; - diagnostics.durable.verifiedPrivateOnlyResponses += - durable.verifiedPrivateOnlyResponses; - diagnostics.durable.dataRejectedMissingMeta += durable.dataRejectedMissingMeta; - diagnostics.durable.rejectedKcs += durable.rejectedKcs; - diagnostics.durable.failedPeers += durable.failedPeers; - diagnostics.durable.failedPhases += durable.failedPhases ?? 0; - diagnostics.durable.deferredBackpressure += durable.deferredBackpressure ?? 0; - deferredBackpressure += durable.deferredBackpressure ?? 0; - diagnostics.durable.deniedPhases = - (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); - peerDenied = peerDenied || durable.deniedPhases > 0; + if (durable) { + dataSynced += durable.insertedDataTriples ?? 0; + diagnostics.durable.fetchedMetaTriples += durable.fetchedMetaTriples; + diagnostics.durable.fetchedDataTriples += durable.fetchedDataTriples; + diagnostics.durable.insertedMetaTriples += durable.insertedMetaTriples; + diagnostics.durable.insertedDataTriples += durable.insertedDataTriples; + diagnostics.durable.bytesReceived += durable.bytesReceived; + diagnostics.durable.resumedPhases += durable.resumedPhases; + diagnostics.durable.timedOutPhases += durable.timedOutPhases ?? 0; + diagnostics.durable.completedPhases += durable.completedPhases ?? 0; + diagnostics.durable.checkpointAdvances += durable.checkpointAdvances ?? 0; + diagnostics.durable.emptyResponses += durable.emptyResponses; + diagnostics.durable.metaOnlyResponses += durable.metaOnlyResponses; + diagnostics.durable.verifiedPrivateOnlyResponses += + durable.verifiedPrivateOnlyResponses; + diagnostics.durable.dataRejectedMissingMeta += durable.dataRejectedMissingMeta; + diagnostics.durable.rejectedKcs += durable.rejectedKcs; + diagnostics.durable.failedPeers += durable.failedPeers; + diagnostics.durable.failedPhases += durable.failedPhases ?? 0; + diagnostics.durable.deferredBackpressure += durable.deferredBackpressure ?? 0; + deferredBackpressure += durable.deferredBackpressure ?? 0; + diagnostics.durable.deniedPhases = + (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); + peerDenied = peerDenied || durable.deniedPhases > 0; - if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { - if ((durable.insertedDataTriples ?? 0) > 0) { - cleanPlaneCompletions.durable.verifiedDataPeers += 1; - } - if (durable.verifiedPrivateOnlyResponses > 0) { - cleanPlaneCompletions.durable.verifiedPrivateOnlyPeers += 1; - } - if ((durable.emptyResponses ?? 0) > 0) { - cleanPlaneCompletions.durable.emptyPeers += 1; + if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { + if ((durable.insertedDataTriples ?? 0) > 0) { + cleanPlaneCompletions.durable.verifiedDataPeers += 1; + } + if (durable.verifiedPrivateOnlyResponses > 0) { + cleanPlaneCompletions.durable.verifiedPrivateOnlyPeers += 1; + } + if ((durable.emptyResponses ?? 0) > 0) { + cleanPlaneCompletions.durable.emptyPeers += 1; + } } } @@ -282,9 +296,54 @@ async function runCatchup(request: CatchupRunRequest): Promise // completed with no timeout. Mirrors the inline // `syncContextGraphFromConnectedPeers` path so both runners report the // same shape. - if (catchupPeerSucceeded(durable, shared, peerDenied, durable.complete)) { + if (catchupPeerSucceeded(durable, shared, peerDenied, durable?.complete)) { peersSucceeded += 1; } + }; + + // Progressive peer walk (issue #2006). The peer list arrives ranked + // authority-first (preferred/curator, then known cores, then the rest), but + // that ordering used never to become *selection*: every sync-capable peer got + // a full durable+SWM pull, so a 14-peer testnet downloaded the same graph 5-6 + // times (147,246 fetched triples for a 24,541-triple graph, ~278MB) and the + // node-wide sync-global queue (2 inflight / 4 queued) saturated against + // itself. + // + // Instead, walk escalating waves (1, 2, 4, ...) and stop as soon as every + // requested plane is proven by real verified data. Wave 1 is the curator when + // one is resolvable, so the happy path transfers exactly one payload. + // + // The stop condition is deliberately POSITIVE-only: an empty round proves + // nothing on its own (an unrelated peer and an empty host are byte-identical + // on the wire), so emptiness stays a whole-round verdict evaluated by + // `catchupPlaneProvenByUnanimousEmpty` after every peer has been walked. + // + // Tradeoff, stated deliberately: a peer's `complete` flag proves it served its + // own manifest, not that the manifest was network-complete. Foreground + // catch-up therefore optimises for one fast authoritative payload; breadth and + // eventual convergence remain the background reconcile lane's job. + // DKG_CATCHUP_STOP_ON_PROOF=0 restores the previous full fan-out. + const waveSizes = CATCHUP_STOP_ON_PROOF + ? catchupWaveSizes(syncCapable.length, CATCHUP_MAX_CONCURRENT_PEER_SYNCS) + : [syncCapable.length]; + let cursor = 0; + for (const waveSize of waveSizes) { + const wave = syncCapable.slice(cursor, cursor + waveSize); + if (wave.length === 0) break; + cursor += wave.length; + peersTried += wave.length; + // Never cancel a wave member mid-stream: an aborted resumed session is + // indistinguishable from a responder supersede. Let the wave finish, then + // stop before dispatching the next one. + const rounds = await mapWithConcurrency( + wave, + Math.min(CATCHUP_MAX_CONCURRENT_PEER_SYNCS, wave.length), + syncPeer, + ); + for (const round of rounds) accumulate(round); + const allRequestedPlanesProven = durableProven() + && (!request.includeSharedMemory || sharedMemoryProven()); + if (CATCHUP_STOP_ON_PROOF && allRequestedPlanesProven) break; } diagnostics.noProtocolPeers = noProtocolPeers; @@ -300,6 +359,7 @@ async function runCatchup(request: CatchupRunRequest): Promise peersTried, peersResponded, peersSucceeded, + peersNotAttempted: syncCapable.length - peersTried, deferredBackpressure, dataSynced, sharedMemorySynced, diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 2aa79a8b22..199416af10 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -37,6 +37,13 @@ export interface CatchupJobResult { * progress or a clean non-metadata-only empty completion. */ peersSucceeded: number; + /** + * Sync-capable peers this run deliberately never contacted because an earlier + * wave already proved every requested plane. These are neither failures nor + * successes; they exist so status mapping and operators can tell an + * early-stopped run from a run where peers were unreachable. + */ + peersNotAttempted?: number; /** Context Graph phases deferred by this node's local sync scheduler. */ deferredBackpressure: number; dataSynced: number; @@ -504,8 +511,94 @@ export function catchupPlaneCompletedWithoutFailure( return classifyDurableProgress(progress, { complete }).completedWithoutFailure; } +/** Per-plane clean-completion evidence accumulated across the peers this run contacted. */ +export interface CatchupPlaneCompletionEvidence { + verifiedDataPeers: number; + /** Peers that cleanly verified one or more V2 KAs with no public triples. */ + verifiedPrivateOnlyPeers?: number; + emptyPeers: number; +} + +/** The aggregate per-plane counters a whole-round verdict is allowed to consult. */ +export interface CatchupPlaneRoundDiagnostics { + fetchedMetaTriples?: number; + fetchedDataTriples?: number; + emptyResponses?: number; + failedPeers?: number; + failedPhases?: number; + timedOutPhases?: number; + deniedPhases?: number; + deferredBackpressure?: number; +} + +/** + * Positive proof: some peer cleanly completed this plane while carrying + * cryptographically verified content. This is the only evidence strong enough + * to stop contacting further peers mid-run, because it is the only evidence a + * single peer can produce on its own. + */ +export function catchupPlaneProvenByData( + completion: CatchupPlaneCompletionEvidence | undefined, +): boolean { + return (completion?.verifiedDataPeers ?? 0) > 0 + || (completion?.verifiedPrivateOnlyPeers ?? 0) > 0; +} + +/** + * Whole-round proof that a public plane really is empty. + * + * A peer that has never heard of a Context Graph and a peer that hosts an empty + * one are byte-identical on the wire: an unknown CG has no access policy, so the + * responder authorizes the request and its CG-scoped queries simply return zero + * rows. The requester only reports `emptyResponses` when BOTH phase payloads are + * empty (`sync-verify-worker-impl.ts`), so an empty response can never carry + * hosting evidence — there is no per-peer signal that could distinguish the two. + * + * Emptiness is therefore only provable as a verdict over the whole round: + * at least one peer completed cleanly empty, nobody delivered any content, and + * nothing failed, timed out, was denied, or was deferred. One data-bearing peer + * that fails is enough to void it — that exact shape (122,705 triples fetched, + * five failed phases, five unrelated peers answering empty) is what settled + * issue #2006's run as `done` with 1 KA out of 40. + */ +export function catchupPlaneProvenByUnanimousEmpty( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + // Empty or metadata-only responses have never been able to prove that a + // private graph is fully synchronized; that stays unchanged. + if (options.isPrivate) return false; + if (catchupPlaneProvenByData(completion)) return false; + const cleanEmptyObserved = (completion?.emptyPeers ?? 0) > 0 + || (diagnostics?.emptyResponses ?? 0) > 0; + if (!cleanEmptyObserved) return false; + if ((diagnostics?.fetchedDataTriples ?? 0) > 0) return false; + if ((diagnostics?.fetchedMetaTriples ?? 0) > 0) return false; + return (diagnostics?.failedPeers ?? 0) === 0 + && (diagnostics?.failedPhases ?? 0) === 0 + && (diagnostics?.timedOutPhases ?? 0) === 0 + && (diagnostics?.deniedPhases ?? 0) === 0 + && (diagnostics?.deferredBackpressure ?? 0) === 0; +} + +/** + * Canonical readiness proof for one catch-up plane. The peer walk stops early + * only on {@link catchupPlaneProvenByData}, so whenever this function falls + * through to the unanimous-empty branch the full peer set really was walked and + * the "nobody saw anything" denominator is meaningful. + */ +export function catchupPlaneReady( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + return catchupPlaneProvenByData(completion) + || catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options); +} + export function catchupPeerSucceeded( - durable: CatchupPhaseProgress, + durable: CatchupPhaseProgress | null | undefined, shared: CatchupPhaseProgress | null | undefined, peerDenied: boolean, durableComplete?: boolean, @@ -533,10 +626,13 @@ export function catchupPeerSucceeded( } export function catchupPeerResponded( - durable: CatchupPhaseProgress, + durable: CatchupPhaseProgress | null | undefined, shared: CatchupPhaseProgress | null | undefined, ): boolean { - const phaseResponded = (phase: CatchupPhaseProgress): boolean => { + // A plane the walk deliberately skipped (already proven by an earlier peer) + // is absent, not silent: it must not be read as this peer having answered. + const phaseResponded = (phase: CatchupPhaseProgress | null | undefined): boolean => { + if (!phase) return false; const progress = classifyDurableProgress(phase); if (progress.transportFailed) return false; if (!progress.deferredByBackpressure) return true; @@ -546,7 +642,7 @@ export function catchupPeerResponded( || (phase.insertedMetaTriples ?? 0) > 0 || (phase.insertedDataTriples ?? phase.insertedTriples ?? 0) > 0; }; - return phaseResponded(durable) || Boolean(shared && phaseResponded(shared)); + return phaseResponded(durable) || phaseResponded(shared); } export interface CatchupRunner { @@ -616,11 +712,25 @@ class WorkerCatchupRunner implements CatchupRunner { } }); this.worker.on('error', (error) => { - for (const [, pending] of this.pendingRuns) pending.reject(error); - this.pendingRuns.clear(); + this.rejectPendingRuns(error); + }); + // `close()` terminates the worker, which emits 'exit' — never 'error'. + // Without this handler every in-flight `run()` promise stays pending + // forever, so the daemon's fire-and-forget subscribe job is pinned at + // `running` with no `finishedAt` for the rest of the process's life. + this.worker.on('exit', (code) => { + this.rejectPendingRuns( + new Error(`Catch-up worker exited (code ${code}) before the run completed`), + ); }); } + private rejectPendingRuns(error: Error): void { + const pending = [...this.pendingRuns.values()]; + this.pendingRuns.clear(); + for (const run of pending) run.reject(error); + } + run(request: CatchupRunRequest): Promise { const runId = this.nextRunId++; return new Promise((resolve, reject) => { @@ -681,22 +791,26 @@ class WorkerCatchupRunner implements CatchupRunner { return agent.waitForSyncProtocol({ toString: () => peerId }); } case 'syncDurable': { - const [peerId, contextGraphId, priority] = args as [string, string, number | undefined]; + const [peerId, contextGraphId, priority, source] = args as [ + string, string, number | undefined, string | undefined, + ]; return agent.syncFromPeerDetailed( peerId, [contextGraphId], undefined, undefined, undefined, - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ); } case 'syncSharedMemory': { - const [peerId, contextGraphId, priority] = args as [string, string, number | undefined]; + const [peerId, contextGraphId, priority, source] = args as [ + string, string, number | undefined, string | undefined, + ]; return agent.syncSharedMemoryFromPeerDetailed( peerId, [contextGraphId], - priority === undefined ? undefined : { priority }, + { ...(priority === undefined ? {} : { priority }), source }, ); } case 'finalizeCatchup': { diff --git a/packages/cli/src/cli-helpers.ts b/packages/cli/src/cli-helpers.ts index 30473b29e3..d572fd5544 100644 --- a/packages/cli/src/cli-helpers.ts +++ b/packages/cli/src/cli-helpers.ts @@ -208,8 +208,10 @@ function printCatchupStatus(status: Awaited 0 ? `, ${notAttempted} not needed` : ''}), data ${status.result.dataSynced}, shared memory ${status.result.sharedMemorySynced}`, ); if (status.result.deferredBackpressure > 0) { console.log(`Deferred: ${status.result.deferredBackpressure} phase(s) by local scheduler backpressure`); diff --git a/packages/cli/src/context-graph-readiness.ts b/packages/cli/src/context-graph-readiness.ts index 18c9e39a4b..874f0a2e3a 100644 --- a/packages/cli/src/context-graph-readiness.ts +++ b/packages/cli/src/context-graph-readiness.ts @@ -6,6 +6,7 @@ import type { } from '@origintrail-official/dkg-node-ui'; import { catchupPlaneCompletedWithoutFailure, + catchupPlaneReady, type CatchupJobResult, } from './catchup-runner.js'; @@ -180,26 +181,28 @@ function catchupPlaneReadyThisRun(input: { plane: 'durable' | 'sharedMemory'; isPrivate: boolean; }): boolean { + const diagnostics = input.result.diagnostics?.[input.plane]; const completion = input.result.cleanPlaneCompletions?.[input.plane]; if (completion) { - const verifiedPrivateOnly = input.plane === 'durable' && - (input.result.cleanPlaneCompletions?.durable.verifiedPrivateOnlyPeers ?? 0) > 0; - return completion.verifiedDataPeers > 0 || - verifiedPrivateOnly || - (!input.isPrivate && completion.emptyPeers > 0); + return catchupPlaneReady(completion, diagnostics, { isPrivate: input.isPrivate }); } // Backward compatibility for callers that construct a legacy result (for // example, an older in-process runner during a rolling upgrade). New worker // results always carry cleanPlaneCompletions, so aggregate failures are not - // used as readiness evidence on the production path. - const diagnostics = input.result.diagnostics?.[input.plane]; + // used as readiness evidence on the production path. The same fail-closed + // rule applies: aggregate counters can show that SOMEBODY answered empty, but + // only a content-free, failure-free round proves the plane really is empty. const dataProgress = input.plane === 'durable' ? input.result.dataSynced > 0 || (input.result.diagnostics?.durable.verifiedPrivateOnlyResponses ?? 0) > 0 : input.result.sharedMemorySynced > 0; - return catchupPlaneCompletedWithoutFailure(diagnostics) && - (dataProgress || (!input.isPrivate && (diagnostics?.emptyResponses ?? 0) > 0)); + if (catchupPlaneCompletedWithoutFailure(diagnostics) && dataProgress) return true; + return catchupPlaneReady( + { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, + diagnostics, + { isPrivate: input.isPrivate }, + ); } export interface ContextGraphCatchupReadinessClassification { diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index c8688098fa..af35ce03a8 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -1,21 +1,34 @@ // catchup-runner-worker-impl.test.ts // -// Drives the daemon-side Worker catch-up implementation — the -// `/api/context-graph/subscribe` path that fans a full durable+SWM sync out -// over every sync-capable peer — over a mocked `parentPort`, and pins the -// 2026-07-07 sync-storm mitigation (C-1) at THIS call site: no more than -// CATCHUP_MAX_CONCURRENT_PEER_SYNCS per-peer sync rounds (or protocol probes) -// may ever be in flight, while every peer still gets synced exactly once, the -// aggregation keeps its one-result-per-peer input-order shape, and one peer's -// failure stays isolated instead of failing the whole run. +// Drives the daemon-side Worker catch-up implementation — the production +// `/api/context-graph/subscribe` path — over a mocked `parentPort`, and pins +// two guarantees at THIS call site: +// +// * the 2026-07-07 sync-storm mitigation (C-1): no more than +// CATCHUP_MAX_CONCURRENT_PEER_SYNCS per-peer sync rounds (or protocol +// probes) may ever be in flight, per-peer failures stay isolated, and the +// aggregation keeps its one-result-per-peer input-order shape; +// * the issue #2006 progressive walk: peers are contacted in escalating +// waves over the authority-ranked list and the walk STOPS as soon as every +// requested plane is proven by verified data, so the happy path transfers +// one payload instead of one per peer. An empty response proves nothing on +// its own and can never stop the walk. import { describe, expect, it, vi } from 'vitest'; import { - CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, CATCHUP_MAX_CONCURRENT_PEER_SYNCS, FOREGROUND_CATCHUP_SYNC_PRIORITY, + catchupWaveSizes, } from '@origintrail-official/dkg-agent'; import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; +// The foreground backpressure budget is wall-clock (default 60s). Shrink it for +// this file so the persistently-deferred case settles quickly; the exact +// deadline arithmetic is pinned deterministically in +// packages/agent/test/catchup-policy.test.ts with an injected clock. +vi.hoisted(() => { + process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = '250'; +}); + // The worker impl wires itself to `parentPort` at module load, so a // controllable port has to be in place BEFORE the module is imported. // Everything else from node:worker_threads stays real. @@ -121,7 +134,74 @@ async function runWorkerCatchup(request: CatchupRunRequest, handler: InvokeHandl } describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1)', () => { - it('caps in-flight peer syncs and protocol probes at the shared limit while still syncing every peer in input order', async () => { + it('stops the walk at the first peer that proves every requested plane', async () => { + const peerIds = Array.from({ length: 20 }, (_, i) => `peer-${i}`); + const durableOrder: string[] = []; + const sharedSeen: string[] = []; + const probeOrder: string[] = []; + const syncPriorities: Array = []; + const syncSources: Array = []; + const finalizeCalls: unknown[][] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-one-payload', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + probeOrder.push(args[0] as string); + return true; + case 'syncDurable': + durableOrder.push(args[0] as string); + syncPriorities.push(args[2] as number | undefined); + syncSources.push(args[3] as string | undefined); + return durableResult(); + case 'syncSharedMemory': + sharedSeen.push(args[0] as string); + syncPriorities.push(args[2] as number | undefined); + syncSources.push(args[3] as string | undefined); + return sharedResult(); + case 'finalizeCatchup': + finalizeCalls.push(args); + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The whole point of issue #2006: one authoritative payload, not twenty. + // Before the progressive walk this was `toEqual(peerIds)` on both planes — + // 20 full durable pulls and 20 full SWM pulls for a single graph. + expect(durableOrder).toEqual(['peer-0']); + expect(sharedSeen).toEqual(['peer-0']); + expect(syncPriorities).toEqual([ + FOREGROUND_CATCHUP_SYNC_PRIORITY, + FOREGROUND_CATCHUP_SYNC_PRIORITY, + ]); + expect(syncSources).toEqual(['catchup-foreground', 'catchup-foreground']); + + // Probing stays eager over the whole connected set: `syncCapablePeers` and + // `noProtocolPeers` are read by daemon status mapping as counts over every + // connected peer, not over the walked prefix. + expect(probeOrder.sort()).toEqual([...peerIds].sort()); + expect(result.syncCapablePeers).toBe(peerIds.length); + expect(result.selectedPeers).toBe(peerIds.length); + + // Peers the walk deliberately skipped are neither tried nor failed. + expect(result.peersTried).toBe(1); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.peersResponded).toBe(1); + expect(result.peersSucceeded).toBe(1); + expect(result.diagnostics?.durable.failedPeers).toBe(0); + expect(result.diagnostics?.durable.timedOutPhases).toBe(0); + + expect(result.deferredBackpressure).toBe(0); + expect(result.dataSynced).toBe(1); + expect(result.sharedMemorySynced).toBe(1); + expect(result.denied).toBe(false); + expect(finalizeCalls).toEqual([['cg-one-payload', 1, 1]]); + }); + + it('escalates waves and still caps in-flight peer syncs when no peer proves the plane', async () => { const peerIds = Array.from({ length: 20 }, (_, i) => `peer-${i}`); // The bound is only observable when the peer set exceeds the cap. expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeLessThan(peerIds.length); @@ -131,11 +211,13 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) let inFlightSyncs = 0; let peakSyncs = 0; const durableOrder: string[] = []; - const sharedSeen: string[] = []; - const syncPriorities: Array = []; - const finalizeCalls: unknown[][] = []; + const startOrder: string[] = []; + + // Nobody completes, so nothing is ever proven and the walk must cover the + // whole peer set — the fallback path. + const unprovenDurable = () => ({ ...durableResult(), complete: false }); - const result = await runWorkerCatchup({ contextGraphId: 'cg-storm', includeSharedMemory: true }, async (method, args) => { + const result = await runWorkerCatchup({ contextGraphId: 'cg-storm', includeSharedMemory: false }, async (method, args) => { switch (method) { case 'prepareCatchup': return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; @@ -148,56 +230,81 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) } case 'syncDurable': { durableOrder.push(args[0] as string); - syncPriorities.push(args[2] as number | undefined); + startOrder.push(args[0] as string); inFlightSyncs += 1; peakSyncs = Math.max(peakSyncs, inFlightSyncs); await delay(4); inFlightSyncs -= 1; - return durableResult(); - } - case 'syncSharedMemory': { - sharedSeen.push(args[0] as string); - syncPriorities.push(args[2] as number | undefined); - inFlightSyncs += 1; - peakSyncs = Math.max(peakSyncs, inFlightSyncs); - await delay(2); - inFlightSyncs -= 1; - return sharedResult(); + return unprovenDurable(); } case 'finalizeCatchup': - finalizeCalls.push(args); return null; default: throw new Error(`unexpected invoke: ${method}`); } }); - // The storm guard: neither fan-out phase ever exceeds the cap… + // The storm guard: neither phase ever exceeds the cap… expect(peakSyncs).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); expect(peakProbes).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); - // …but the fan-out is still actually parallel, not accidentally serialised. + // …but later waves are still actually parallel, not accidentally serialised. expect(peakSyncs).toBeGreaterThan(1); + // The first wave is a single peer, so a proving authority costs one payload. + expect(catchupWaveSizes(peerIds.length, CATCHUP_MAX_CONCURRENT_PEER_SYNCS)[0]).toBe(1); + expect(startOrder[0]).toBe('peer-0'); - // Coverage preserved: every peer synced exactly once, started in input - // order (the bounded mapper's shared cursor hands out work in order). + // Coverage preserved when nothing proves: every peer walked, in rank order. expect(durableOrder).toEqual(peerIds); - expect([...sharedSeen].sort()).toEqual([...peerIds].sort()); - expect(syncPriorities).toEqual( - Array.from({ length: peerIds.length * 2 }, () => FOREGROUND_CATCHUP_SYNC_PRIORITY), - ); - - // Aggregation unchanged from the unbounded Promise.all shape. - expect(result.selectedPeers).toBe(peerIds.length); - expect(result.syncCapablePeers).toBe(peerIds.length); expect(result.peersTried).toBe(peerIds.length); - expect(result.peersResponded).toBe(peerIds.length); - expect(result.peersSucceeded).toBe(peerIds.length); - expect(result.deferredBackpressure).toBe(0); + expect(result.peersNotAttempted).toBe(0); expect(result.dataSynced).toBe(peerIds.length); - expect(result.sharedMemorySynced).toBe(peerIds.length); - expect(result.denied).toBe(false); - expect(result.diagnostics?.durable.failedPeers).toBe(0); - expect(finalizeCalls).toEqual([['cg-storm', peerIds.length, peerIds.length]]); + }); + + it('narrows fallback peers to the planes still in question', async () => { + // A Context Graph whose public durable data is empty can never prove its + // durable plane by verified data, so the walk must cover every peer to + // establish the whole-round empty verdict. Without per-plane narrowing that + // would drag a full re-pull of the ALREADY PROVEN shared-memory plane out of + // every remaining peer — the exact amplification this fix removes. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-only', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Shared memory is proven by the first peer and never pulled again… + expect(sharedCalls).toEqual(['peer-0']); + // …while the (cheap, empty) durable plane still walks everyone, because a + // clean empty answer only proves emptiness as a whole-round verdict. + expect(durableCalls).toEqual(peerIds); + expect(result.sharedMemorySynced).toBe(1); + expect(result.cleanPlaneCompletions?.sharedMemory.verifiedDataPeers).toBe(1); + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(peerIds.length); }); it('keeps per-peer failure isolation and probe filtering under the bounded fan-out', async () => { @@ -218,7 +325,10 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) durableCalls.push(args[0] as string); await delay(2); if (args[0] === 'peer-3') throw new Error('peer 3 exploded'); - return durableResult(); + // `complete: false` keeps every peer unproven, so the walk covers the + // whole set and the isolation claim below is actually exercised + // instead of being skipped by an early stop. + return { ...durableResult(), complete: false }; } case 'syncSharedMemory': sharedCalls += 1; @@ -235,6 +345,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(sharedCalls).toBe(0); expect(result.syncCapablePeers).toBe(11); expect(result.peersTried).toBe(11); + expect(result.peersNotAttempted).toBe(0); expect(result.peersResponded).toBe(10); expect(result.peersSucceeded).toBe(10); expect(result.dataSynced).toBe(10); @@ -243,6 +354,61 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.diagnostics?.durable.failedPeers).toBe(1); }); + it('does not let an unrelated peer\'s clean empty response stop the walk or prove readiness', async () => { + // The reported #2006 shape: a data-bearing peer fails part-way, an + // unrelated peer that has never heard of the graph answers empty. On the + // wire those two peers are indistinguishable, so the empty answer must not + // stop the walk and must not settle the job as `done`. + const peerIds = ['peer-empty', 'peer-data-failed', 'peer-quiet']; + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-mask', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + if (args[0] === 'peer-data-failed') { + return { + ...durableResult(), + complete: false, + fetchedDataTriples: 5_000, + insertedTriples: 0, + insertedDataTriples: 0, + completedPhases: 0, + timedOutPhases: 1, + failedPhases: 1, + }; + } + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Emptiness is never a stop condition, so every peer is still contacted. + expect(durableCalls).toEqual(peerIds); + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBe(0); + // The clean-empty peers are still recorded as clean empty completions… + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(2); + // …but the round fetched data and failed, so readiness must not follow. + expect(result.diagnostics?.durable.fetchedDataTriples).toBe(5_000); + expect(result.diagnostics?.durable.failedPhases).toBe(1); + }); + it('retries only SWM after durable progress and finalizes when local pressure clears', async () => { const finalizeCalls: unknown[][] = []; let durableCalls = 0; @@ -332,7 +498,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(callOrder).toEqual(['durable-1', 'durable-2', 'shared']); }); - it('returns deferred after a bounded durable retry budget and never starts dependent SWM', async () => { + it('returns deferred after the wall-clock durable retry budget and never starts dependent SWM', async () => { let durableCalls = 0; let sharedCalls = 0; const finalizeCalls: unknown[][] = []; @@ -368,7 +534,12 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }, ); - expect(durableCalls).toBe(CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS.length + 1); + // The retry loop is wired and bounded: at least one retry happened (the + // pre-#2006 policy also retried, but on a fixed 850 ms ladder), and the run + // settled at the wall-clock budget instead of spinning forever. The exact + // deadline arithmetic — that attempts are governed by the clock, not by a + // fixed attempt count — is pinned in packages/agent/test/catchup-policy.test.ts. + expect(durableCalls).toBeGreaterThanOrEqual(2); expect(sharedCalls).toBe(0); expect(result.deferredBackpressure).toBe(1); expect(result.peersResponded).toBe(0); @@ -425,10 +596,14 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) async (method, args) => { switch (method) { case 'prepareCatchup': + // The denying/timing-out peer is ranked FIRST so the walk reaches + // the clean peer in a later wave: the claim under test is that a + // clean per-peer completion survives another peer's denial in the + // aggregate, which an early stop on wave 1 would never exercise. return { preferredPeerId: undefined, isPrivateContextGraph: true, - peerIds: ['peer-clean', 'peer-partial'], + peerIds: ['peer-partial', 'peer-clean'], connectedPeers: 2, }; case 'waitForSyncProtocol': @@ -585,6 +760,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }); }); + // A clean empty round is never a stop condition — it cannot distinguish an + // empty host from a peer that never heard of the graph — so every peer is + // still walked and emptiness stays a whole-round verdict. it('records each distinct responder that cleanly completes both planes empty', async () => { const peerIds = ['peer-empty-1', 'peer-empty-2', 'peer-empty-3', 'peer-empty-4']; const result = await runWorkerCatchup( @@ -631,6 +809,8 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result).toMatchObject({ peersResponded: peerIds.length, peersSucceeded: peerIds.length, + peersTried: peerIds.length, + peersNotAttempted: 0, dataSynced: 0, sharedMemorySynced: 0, }); diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts new file mode 100644 index 0000000000..4ea608ee30 --- /dev/null +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -0,0 +1,38 @@ +// The daemon's subscribe job awaits `catchupRunner.run(...)` in a +// fire-and-forget async IIFE with no timeout of its own. `close()` terminates +// the Worker, which emits 'exit' — never 'error' — so before this fix a pending +// run promise was simply never settled and the job stayed `running` with no +// `finishedAt` for the rest of the process's life. Issue #2006 makes that +// reachable routinely, because a walk can be in flight for much longer. +import { describe, expect, it } from 'vitest'; +import type { DKGAgent } from '@origintrail-official/dkg-agent'; +import { createCatchupRunner } from '../src/catchup-runner.js'; + +function hangingAgent(): DKGAgent { + return { + isPrivateContextGraph: async () => false, + resolvePreferredSyncPeerId: async () => undefined, + ensurePeerConnected: async () => {}, + // The worker's first RPC never settles, so the run stays in `pendingRuns`. + primeCatchupConnections: () => new Promise(() => {}), + selectCatchupPeers: (peers: unknown[]) => peers, + node: { libp2p: { getConnections: () => [] } }, + } as unknown as DKGAgent; +} + +describe('WorkerCatchupRunner lifecycle', () => { + it('rejects an in-flight run when the worker exits instead of leaving it pending forever', async () => { + const runner = createCatchupRunner(hangingAgent()); + const run = runner.run({ contextGraphId: 'cg-hang', includeSharedMemory: false }); + const settled = run.then(() => 'resolved' as const, () => 'rejected' as const); + + // Give the worker a moment to boot and issue its first invoke. + await new Promise((resolve) => setTimeout(resolve, 200)); + await runner.close(); + + await expect(Promise.race([ + settled, + new Promise((resolve) => setTimeout(() => resolve('pending'), 5_000)), + ])).resolves.toBe('rejected'); + }, 30_000); +}); diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index adf7558284..b86e70d90b 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -3,6 +3,9 @@ import { catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByData, + catchupPlaneProvenByUnanimousEmpty, + catchupPlaneReady, classifyDurableCatchupRequest, runDurableCatchupLeg, } from '../src/catchup-runner.js'; @@ -635,3 +638,100 @@ describe('route-level durable catchup orchestration', () => { }); }); }); + +// Issue #2006. On the wire, a peer that hosts an empty Context Graph and a peer +// that has never heard of it are byte-identical: an unknown CG has no access +// policy, so the responder authorizes the request and its CG-scoped queries +// simply return zero rows. The requester emits `emptyResponses` only when BOTH +// phase payloads are empty, so an empty response can never carry hosting +// evidence. Emptiness is therefore only provable as a whole-round verdict. +describe('catch-up plane proof predicates', () => { + const noEvidence = { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }; + const cleanEmptyRound = { + fetchedMetaTriples: 0, + fetchedDataTriples: 0, + emptyResponses: 2, + failedPeers: 0, + failedPhases: 0, + timedOutPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; + const emptyPeers = { ...noEvidence, emptyPeers: 2 }; + + it('treats verified data and verified private-only completions as positive proof', () => { + expect(catchupPlaneProvenByData({ ...noEvidence, verifiedDataPeers: 1 })).toBe(true); + expect(catchupPlaneProvenByData({ ...noEvidence, verifiedPrivateOnlyPeers: 1 })).toBe(true); + expect(catchupPlaneProvenByData(noEvidence)).toBe(false); + expect(catchupPlaneProvenByData(undefined)).toBe(false); + // A clean empty response is NOT positive proof — it can never stop the walk. + expect(catchupPlaneProvenByData(emptyPeers)).toBe(false); + }); + + it('accepts a unanimously clean, content-free public round as proof of emptiness', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, cleanEmptyRound, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, cleanEmptyRound, { isPrivate: false })).toBe(true); + }); + + it('never proves a private plane from an empty round', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, cleanEmptyRound, { isPrivate: true })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, cleanEmptyRound, { isPrivate: true })).toBe(false); + }); + + it.each([ + ['a data-bearing peer that failed', { fetchedDataTriples: 122_705, failedPhases: 5 }], + ['fetched data with no verified completion', { fetchedDataTriples: 5_000 }], + ['a peer that returned metadata', { fetchedMetaTriples: 12 }], + ['a transport failure', { failedPeers: 1 }], + ['a failed phase', { failedPhases: 1 }], + ['a timed-out phase', { timedOutPhases: 1 }], + ['a denial', { deniedPhases: 1 }], + ['a local admission deferral', { deferredBackpressure: 1 }], + ])('voids the empty proof when the round contains %s', (_label, overrides) => { + const diagnostics = { ...cleanEmptyRound, ...overrides }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(false); + }); + + it('still reports ready when a peer delivered verified data despite other failures', () => { + const diagnostics = { ...cleanEmptyRound, fetchedDataTriples: 24_541, failedPhases: 1 }; + const completion = { ...emptyPeers, verifiedDataPeers: 1 }; + expect(catchupPlaneReady(completion, diagnostics, { isPrivate: false })).toBe(true); + expect(catchupPlaneReady(completion, diagnostics, { isPrivate: true })).toBe(true); + }); + + it('requires at least one clean empty completion before an empty verdict', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + noEvidence, + { ...cleanEmptyRound, emptyResponses: 0 }, + { isPrivate: false }, + )).toBe(false); + }); +}); + +describe('catch-up peer accounting with a skipped plane', () => { + const cleanShared = { + insertedTriples: 3, + insertedDataTriples: 3, + completedPhases: 1, + bytesReceived: 30, + }; + + it('does not read a skipped durable plane as a peer response', () => { + // The walk omits the durable plane for peers contacted purely as a + // shared-memory fallback. An absent plane is not a silent one: it must not + // manufacture a response for a peer whose only requested plane failed. + expect(catchupPeerResponded(null, { failedPeers: 1 })).toBe(false); + expect(catchupPeerResponded(null, undefined)).toBe(false); + expect(catchupPeerResponded(null, cleanShared)).toBe(true); + }); + + it('judges a skipped durable plane purely on the shared-memory outcome', () => { + expect(catchupPeerSucceeded(null, cleanShared, false)).toBe(true); + expect(catchupPeerSucceeded(null, { ...cleanShared, timedOutPhases: 1 }, false)).toBe(false); + expect(catchupPeerSucceeded(null, { failedPeers: 1 }, false)).toBe(false); + }); +}); diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 3378796855..61f10a507b 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -186,20 +186,32 @@ describe('context graph catch-up readiness classification', () => { expect(classification.eventPayload).toBeUndefined(); }); - it('accepts a public clean-empty peer when another peer denies', () => { + // Emptiness is only provable as a whole-round verdict: an empty response is + // byte-identical whether the peer hosts an empty graph or has never heard of + // it, so a clean-empty peer proves the plane only when NOBODY in the round + // delivered content and nothing failed. + function publicEmptyRoundResult(): CatchupJobResult { const result = mixedPeerResult(0); result.dataSynced = 0; - result.peersSucceeded = 1; + result.peersSucceeded = 2; + result.denied = false; + result.deniedPeers = 0; if (!result.cleanPlaneCompletions || !result.diagnostics?.durable) { throw new Error('durable completion evidence missing'); } - result.cleanPlaneCompletions.durable.emptyPeers = 1; + result.cleanPlaneCompletions.durable.emptyPeers = 2; result.diagnostics.durable.fetchedDataTriples = 0; result.diagnostics.durable.insertedDataTriples = 0; - result.diagnostics.durable.emptyResponses = 1; + result.diagnostics.durable.emptyResponses = 2; + result.diagnostics.durable.timedOutPhases = 0; + result.diagnostics.durable.deniedPhases = 0; + result.diagnostics.durable.completedPhases = 4; + return result; + } + it('accepts a unanimously clean-empty public round as proof the plane is empty', () => { const classification = classifyContextGraphCatchupReadiness({ - result, + result: publicEmptyRoundResult(), includeSharedMemory: false, hasConfirmedMeta: true, isPrivate: false, @@ -218,4 +230,63 @@ describe('context graph catch-up readiness classification', () => { }, }); }); + + it('does not accept a public clean-empty peer when another peer denies', () => { + // A denial means we did not hear from every peer, so "nobody has anything" + // is not established. Before #2006 this returned `done`. + const result = publicEmptyRoundResult(); + result.denied = true; + result.deniedPeers = 1; + result.cleanPlaneCompletions!.durable.emptyPeers = 1; + result.diagnostics!.durable.emptyResponses = 1; + result.diagnostics!.durable.deniedPhases = 1; + + const classification = classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification).toMatchObject({ + jobStatus: 'unreachable', + readinessPatch: { durableVerified: false, sharedMemoryVerified: false }, + }); + }); + + it('does not let a clean-empty peer mask a data-bearing peer that failed', () => { + // The exact reported #2006 shape: 122,705 triples fetched, five phases + // failed, no verified data completion, and unrelated peers answering empty. + const result = publicEmptyRoundResult(); + result.cleanPlaneCompletions!.durable.emptyPeers = 1; + result.diagnostics!.durable.emptyResponses = 1; + result.diagnostics!.durable.fetchedDataTriples = 122_705; + result.diagnostics!.durable.failedPhases = 5; + + const classification = classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + }); + + it('never proves a private plane from an empty round', () => { + const classification = classifyContextGraphCatchupReadiness({ + result: publicEmptyRoundResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: true, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).toBe('unreachable'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + }); }); diff --git a/packages/cli/test/context-graph-subscribe-readiness.test.ts b/packages/cli/test/context-graph-subscribe-readiness.test.ts index 6eac7e0137..ad47313dd5 100644 --- a/packages/cli/test/context-graph-subscribe-readiness.test.ts +++ b/packages/cli/test/context-graph-subscribe-readiness.test.ts @@ -528,7 +528,11 @@ describe('context graph subscribe readiness requires authoritative metadata', () }); }); - it('keeps a public clean-empty peer valid when another peer denies', async () => { + // Issue #2006: an empty response cannot distinguish "hosts an empty graph" + // from "never heard of this graph", so a clean-empty peer only proves the + // plane when the whole round was content-free and failure-free. A denial or a + // failed data-bearing peer means we did not hear from everyone. + it('does not keep a public clean-empty peer valid when another peer denies', async () => { const mixed = cleanEmptyResult(); mixed.connectedPeers = 2; mixed.totalPeers = 2; @@ -553,19 +557,47 @@ describe('context graph subscribe readiness requires authoritative metadata', () }, }); - expect(result.job.status).toBe('done'); - expect(result.job.error).toBeUndefined(); - expect(result.state).toMatchObject({ - synced: true, - sharedMemorySynced: false, - metaSynced: true, - }); + expect(result.job.status).not.toBe('done'); + expect(result.job.status).toBe('unreachable'); + expect(result.state).toMatchObject({ synced: false }); expect(result.readiness).toMatchObject({ - durableVerified: true, + durableVerified: false, sharedMemoryVerified: false, }); }); + it('does not settle as done when a data-bearing peer failed and an unrelated peer answered empty', async () => { + // The reported field shape: 122,705 data triples fetched, five failed + // phases, nothing verified, and unrelated peers answering empty — which + // previously settled the job as `done` with 1 KA out of 40. + const masked = cleanEmptyResult(); + masked.connectedPeers = 6; + masked.totalPeers = 6; + masked.selectedPeers = 6; + masked.syncCapablePeers = 6; + masked.peersTried = 6; + masked.peersResponded = 6; + if (!masked.diagnostics?.durable) throw new Error('durable diagnostics missing'); + masked.diagnostics.durable.fetchedDataTriples = 122_705; + masked.diagnostics.durable.failedPhases = 5; + + const result = await subscribe({ + hasConfirmedMeta: true, + includeSharedMemory: false, + result: masked, + initial: { + subscribed: true, + synced: false, + sharedMemorySynced: false, + metaSynced: true, + }, + }); + + expect(result.job.status).not.toBe('done'); + expect(result.state).toMatchObject({ synced: false }); + expect(result.readiness).toMatchObject({ durableVerified: false }); + }); + it('does not promote private data readiness from unrelated empty responders after metadata is local', async () => { const result = await subscribe({ hasConfirmedMeta: true, diff --git a/packages/cli/vitest.unit.config.ts b/packages/cli/vitest.unit.config.ts index c1dee3b3fe..cdca0b4175 100644 --- a/packages/cli/vitest.unit.config.ts +++ b/packages/cli/vitest.unit.config.ts @@ -67,6 +67,7 @@ export default defineConfig({ 'test/random-sampling-status.test.ts', 'test/catchup-runner.test.ts', 'test/catchup-runner-worker-impl.test.ts', + 'test/catchup-runner-worker-lifecycle.test.ts', 'test/relay-status-block.test.ts', 'test/supervisor-liveness.test.ts', 'test/promote-async-routes.test.ts', From 76b16a2db1637083a3a3ec347d61d519bb098733 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 03:06:45 +0200 Subject: [PATCH 02/44] fix(sync): spend the single-peer opening wave only on a resolvable curator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A single-peer first wave buys "one payload from the authority". When no curator resolves — or the resolved one is not sync-capable — it buys nothing and just prepends a serial round-trip to every catch-up round, which a live testnet run made visible as a materially longer subscribe for a graph no connected peer hosted. The walk now opens at the full concurrency cap in that case, keeping the previous first-round latency while still stopping as soon as a plane is proven. `catchupWaveSizes` takes an explicit `startWidth`, clamped to the cap. Co-Authored-By: Claude Opus 5 (1M context) --- .../agent/src/sync/catchup-concurrency.ts | 26 +++++--- .../agent/test/catchup-concurrency.test.ts | 11 ++++ .../cli/src/catchup-runner-worker-impl.ts | 15 ++++- .../test/catchup-runner-worker-impl.test.ts | 64 +++++++++++++++++++ 4 files changed, 106 insertions(+), 10 deletions(-) diff --git a/packages/agent/src/sync/catchup-concurrency.ts b/packages/agent/src/sync/catchup-concurrency.ts index 8bb916bcea..b86b6ed738 100644 --- a/packages/agent/src/sync/catchup-concurrency.ts +++ b/packages/agent/src/sync/catchup-concurrency.ts @@ -19,20 +19,28 @@ export const CATCHUP_STOP_ON_PROOF: boolean = (() => { })(); /** - * Escalating wave sizes for the progressive peer walk: 1, 2, 4, … capped by - * `maxConcurrency` and truncated to `peerCount`. + * Escalating wave sizes for the progressive peer walk: `startWidth`, ×2, ×2, … + * capped by `maxConcurrency` and truncated to `peerCount`. * - * The first wave is a single peer because the peer list is already ranked - * authority-first (preferred/curator, then known cores), so the happy path - * downloads exactly one payload. Doubling afterwards keeps the fallback tail - * short — a flat wave of `maxConcurrency` would pull that many concurrent full - * payloads before any of them could prove the plane. + * With `startWidth = 1` the first wave is a single peer, which is what makes an + * authoritative first peer cost exactly one payload; doubling afterwards keeps + * the fallback tail short, because a flat wave of `maxConcurrency` would pull + * that many concurrent full payloads before any of them could prove the plane. + * + * A single-peer first wave is only justified when there IS an authority to try + * first. With no resolvable curator the head of the ranked list has no special + * claim, so callers pass `startWidth = maxConcurrency` and the walk keeps the + * previous first-round latency while still stopping early on proof. */ -export function catchupWaveSizes(peerCount: number, maxConcurrency: number): number[] { +export function catchupWaveSizes( + peerCount: number, + maxConcurrency: number, + startWidth = 1, +): number[] { const cap = Number.isInteger(maxConcurrency) && maxConcurrency > 0 ? maxConcurrency : 1; const sizes: number[] = []; let remaining = Math.max(0, Math.trunc(peerCount)); - let size = 1; + let size = Number.isInteger(startWidth) && startWidth > 0 ? Math.min(cap, startWidth) : 1; while (remaining > 0) { const take = Math.min(size, remaining); sizes.push(take); diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts index d2a43a4144..dc5c749955 100644 --- a/packages/agent/test/catchup-concurrency.test.ts +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -19,6 +19,17 @@ describe('catchupWaveSizes', () => { expect(catchupWaveSizes(7, 8)).toEqual([1, 2, 4]); }); + it('opens at the full cap when there is no authority to spend the first wave on', () => { + // A single-peer opening wave buys "one payload from the curator". With no + // resolvable curator it buys nothing and would just add a round-trip to the + // front of every round, so callers open at the cap instead. + expect(catchupWaveSizes(14, 4, 4)).toEqual([4, 4, 4, 2]); + expect(catchupWaveSizes(3, 4, 4)).toEqual([3]); + // startWidth can never exceed the concurrency cap. + expect(catchupWaveSizes(9, 2, 8)).toEqual([2, 2, 2, 2, 1]); + expect(catchupWaveSizes(5, 4, 0)).toEqual([1, 2, 2]); + }); + it('never exceeds the cap or the peer count', () => { for (const cap of [1, 2, 3, 4, 8]) { for (const peerCount of [0, 1, 3, 5, 13, 40]) { diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index c064794b02..914dfd30a8 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -323,8 +323,21 @@ async function runCatchup(request: CatchupRunRequest): Promise // catch-up therefore optimises for one fast authoritative payload; breadth and // eventual convergence remain the background reconcile lane's job. // DKG_CATCHUP_STOP_ON_PROOF=0 restores the previous full fan-out. + // + // The single-peer opening wave is spent on the authority, so it is only taken + // when there IS one: if no curator resolved (or it is not sync-capable), the + // head of the ranked list has no special claim and serialising it would just + // add a round-trip to the front of every round. In that case the walk opens at + // the full concurrency cap — the previous first-round latency — and still + // stops as soon as something is proven. + const authorityFirst = prepared.preferredPeerId !== undefined + && syncCapable[0] === prepared.preferredPeerId; const waveSizes = CATCHUP_STOP_ON_PROOF - ? catchupWaveSizes(syncCapable.length, CATCHUP_MAX_CONCURRENT_PEER_SYNCS) + ? catchupWaveSizes( + syncCapable.length, + CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + authorityFirst ? 1 : CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + ) : [syncCapable.length]; let cursor = 0; for (const waveSize of waveSizes) { diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index af35ce03a8..ccc6caed72 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -260,6 +260,70 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.dataSynced).toBe(peerIds.length); }); + it('opens at the full concurrency cap when no curator resolved', async () => { + // A single-peer opening wave buys "one payload from the curator". Without a + // resolvable curator it buys nothing, so the walk must not serialise the + // head of the list and pay an extra round-trip on every round. + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + let inFlight = 0; + let peak = 0; + const startOrder: string[] = []; + + await runWorkerCatchup({ contextGraphId: 'cg-no-curator', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + startOrder.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(4); + inFlight -= 1; + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(startOrder).toEqual(peerIds); + }); + + it('spends the single-peer opening wave only on a sync-capable curator', async () => { + // The curator is ranked first but is NOT sync-capable, so the walk has no + // authority to try alone and must not serialise an arbitrary peer instead. + const peerIds = ['peer-curator', 'peer-a', 'peer-b', 'peer-c', 'peer-d']; + let inFlight = 0; + let peak = 0; + + await runWorkerCatchup({ contextGraphId: 'cg-curator-offline', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-curator', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return args[0] !== 'peer-curator'; + case 'syncDurable': { + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(4); + inFlight -= 1; + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + }); + it('narrows fallback peers to the planes still in question', async () => { // A Context Graph whose public durable data is empty can never prove its // durable plane by verified data, so the walk must cover every peer to From d3317b0bf91e895ddfcdbd1902dc8292b2a7da77 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 03:31:25 +0200 Subject: [PATCH 03/44] fix(sync): gate the catch-up early stop on authoritative proof (review round 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 1 on #2007. Three findings applied, four answered in thread. * Only the resolved curator's snapshot is a reference for the WHOLE graph. A peer's `complete` flag proves it served its OWN manifest, so a non-authoritative peer holding a subset could previously cut the walk short and strand another peer's Knowledge Assets. Both optimisations — skipping remaining peers and skipping an already-settled plane on the peers still contacted — are now gated on AUTHORITY proof. With no resolvable curator the walk degrades to the previous full bounded fan-out and keeps unioning every peer's data. Adds a regression test with two clean peers carrying disjoint data. * The backpressure budget default was 60s, below both the 87-109s queue waits the change exists to survive and the 120s head-of-line round that produces them. Raised to 180s, with a virtual-clock test that keeps retrying past a 90s capacity clear and a test asserting the constant stays above both numbers. * The kill-switch had no automated coverage. Adds a dedicated test file that boots the worker with DKG_CATCHUP_STOP_ON_PROOF=0 and asserts the full fan-out is restored — every peer, both planes, still under the sync-storm concurrency bound — while a curator that proves both planes on wave 1 no longer stops the run. * `runContextGraphSyncWithBackpressure` took three optional positional tail parameters, so call sites read `undefined, undefined, 'swm-recovery'`. Replaced with a named `admission` object. Co-Authored-By: Claude Opus 5 (1M context) --- packages/agent/src/dkg-agent-lifecycle.ts | 31 ++-- packages/agent/src/sync/catchup-policy.ts | 14 +- packages/agent/test/catchup-policy.test.ts | 32 ++++ .../cli/src/catchup-runner-worker-impl.ts | 57 +++--- .../test/catchup-runner-worker-impl.test.ts | 30 ++++ .../catchup-runner-worker-killswitch.test.ts | 164 ++++++++++++++++++ packages/cli/vitest.unit.config.ts | 1 + 7 files changed, 287 insertions(+), 42 deletions(-) create mode 100644 packages/cli/test/catchup-runner-worker-killswitch.test.ts diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 1adcfd049b..048517c592 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -1159,10 +1159,15 @@ export class LifecycleSyncMethods extends DKGAgentBase { lane: SyncSchedulerLane, label: string, work: () => Promise, - priorityOverride?: number, - operationSignal?: AbortSignal, - source?: string, + admission: { + /** Admission override for foreground catch-up / VM recovery. */ + priorityOverride?: number; + operationSignal?: AbortSignal; + /** Which trigger enqueued this admission; clamped to the closed set. */ + source?: string; + } = {}, ): Promise { + const { priorityOverride, operationSignal, source } = admission; const priority = priorityOverride ?? contextGraphPriority(this.config.syncContextGraphPriorities, contextGraphId); const admissionBoundary = combineSyncAdmissionSignals( @@ -4471,9 +4476,11 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, work, - options?.priority, - operationBoundary.signal, - options?.source, + { + priorityOverride: options?.priority, + operationSignal: operationBoundary.signal, + source: options?.source, + }, ), operationBoundary.signal, ); @@ -4792,9 +4799,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, run, - priority, - undefined, - source, + { priorityOverride: priority, source }, ), merge: mergeDurableSyncAccumulatorInto, markDeferred: (summary) => { @@ -5456,9 +5461,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { item.lane, item.operationId, run, - options?.priority, - undefined, - options?.source, + { priorityOverride: options?.priority, source: options?.source }, ), merge: mergeSharedMemorySyncResults, markDeferred: (summary) => ({ @@ -5542,9 +5545,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { remotePeerId, contextGraphId, ), - undefined, - undefined, - 'swm-recovery', + { source: 'swm-recovery' }, ); } diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 920e4b8c16..dc5728987f 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -16,14 +16,18 @@ export const CATCHUP_BACKPRESSURE_JITTER_RATIO = 0.25; * while an admitted `sync-global` round is bounded by `SYNC_TOTAL_TIMEOUT_MS` * (120 s) per plane, and issue #2006 measured queue waits of 87–109 s. A refused * foreground admission therefore always exhausted its budget long before the - * head of the queue could possibly have cleared. Waiting costs a timer and no - * work, so the budget is now wall-clock and generous — but still bounded, so a - * permanently saturated node fails the catch-up job instead of pinning it at - * `running` forever. + * head of the queue could possibly have cleared. + * + * The default is deliberately set ABOVE both of those numbers: the wait has to + * outlast one full head-of-line round (120 s) plus the observed backlog, or the + * budget still gives up in exactly the saturation case it exists to survive. + * Waiting costs a timer and no work. It stays bounded, so a permanently + * saturated node fails the catch-up job with a retryable status instead of + * pinning it at `running` forever. */ export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = (() => { const raw = Number(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); - return Number.isInteger(raw) && raw >= 0 ? raw : 60_000; + return Number.isInteger(raw) && raw >= 0 ? raw : 180_000; })(); /** Bounded admission origin recorded on node-wide scheduler diagnostics. */ diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index f4eeb84925..d2f684ea17 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest'; import { CATCHUP_BACKPRESSURE_BASE_DELAY_MS, CATCHUP_BACKPRESSURE_MAX_DELAY_MS, + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, FOREGROUND_CATCHUP_SYNC_PRIORITY, nextCatchupBackpressureDelayMs, runCatchupPlaneWithPolicy, @@ -235,3 +236,34 @@ describe('nextCatchupBackpressureDelayMs', () => { expect(nextCatchupBackpressureDelayMs({ attempt: 0, remainingMs: -5 })).toBeUndefined(); }); }); + +describe('CATCHUP_BACKPRESSURE_MAX_WAIT_MS', () => { + it('outlasts one head-of-line round plus the queue waits it exists to survive', () => { + // Issue #2006 measured `sync-global` queue waits of 87-109 s, and an + // admitted round is itself bounded by SYNC_TOTAL_TIMEOUT_MS (120 s). A + // budget below those numbers gives up in exactly the saturation case it was + // introduced for — which is what the old fixed 850 ms ladder did. + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBeGreaterThan(120_000); + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBeGreaterThan(109_000); + }); + + it('keeps retrying past a 90-second capacity clear under the default budget', async () => { + // Virtual clock: admission stays refused until 90 s have elapsed, i.e. a + // realistic head-of-line drain. The default policy must still be retrying + // then, and must succeed rather than return deferred. + const clock = virtualClock(0); + const syncDurable = vi.fn(async () => ( + clock.now() >= 90_000 ? { deferredBackpressure: 0 } : { deferredBackpressure: 1 } + )); + + const result = await runCatchupPlaneWithPolicy('foreground', syncDurable, { + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + expect(result.deferredBackpressure).toBe(0); + expect(clock.now()).toBeGreaterThanOrEqual(90_000); + expect(clock.now()).toBeLessThanOrEqual(CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); +}); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 914dfd30a8..dd7b3943ca 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -12,7 +12,6 @@ import { catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, - catchupPlaneProvenByData, type CatchupJobResult, type CatchupRunRequest, } from './catchup-runner.js'; @@ -54,8 +53,11 @@ parentPort!.on('message', async (message: any) => { } }); -/** A per-peer sync round; `durable` is absent when the walk skipped that plane. */ +/** A per-peer sync round; a plane is absent when the walk skipped it. */ interface PeerRound { + peerId: string; + /** The resolved curator for this Context Graph produced this round. */ + fromAuthority: boolean; durable: any | null; shared: any | null; } @@ -168,9 +170,17 @@ async function runCatchup(request: CatchupRunRequest): Promise } syncCapablePeers = syncCapable.length; - const durableProven = (): boolean => catchupPlaneProvenByData(cleanPlaneCompletions.durable); - const sharedMemoryProven = (): boolean => - catchupPlaneProvenByData(cleanPlaneCompletions.sharedMemory); + // Only the resolved curator's snapshot is a reference for the WHOLE graph: a + // peer's `complete` flag proves it served its own manifest, so a + // non-authoritative peer carrying a subset would otherwise be able to cut the + // walk short and strand another peer's Knowledge Assets. Both optimisations + // below — skipping remaining peers, and skipping an already-proven plane on + // the peers we do contact — are therefore gated on AUTHORITY proof. With no + // resolvable curator the walk degrades to the previous full bounded fan-out + // and keeps unioning every peer's data. + const authorityProven = { durable: false, sharedMemory: false }; + const authorityProvedEverything = (): boolean => authorityProven.durable + && (!request.includeSharedMemory || authorityProven.sharedMemory); // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we // can from the other peers instead of failing the entire subscribe/catch-up. @@ -186,33 +196,34 @@ async function runCatchup(request: CatchupRunRequest): Promise invoke('syncSharedMemory', peerId, request.contextGraphId, priority, source) .catch(() => emptyShared()); - // Narrow each fallback peer to the planes still in question. The walk only - // continues while some requested plane is unproven, and one plane is often - // proven long before the other — a Context Graph whose public VM data is - // empty can never prove its durable plane by data, so without this a single - // unproven plane would drag a full re-pull of the ALREADY PROVEN plane out - // of every remaining peer, which is the amplification this fix exists to - // remove. - // The kill-switch restores the previous fan-out faithfully: every peer, both - // requested planes, no early stop. - const needDurable = !CATCHUP_STOP_ON_PROOF || !durableProven(); + // Narrow each fallback peer to the planes the AUTHORITY has not already + // settled. One plane is often settled long before the other — a Context + // Graph whose public VM data is empty can never prove its durable plane by + // data — so without this a single unproven plane would drag a full re-pull + // of the already-settled plane out of every remaining peer, which is the + // amplification this fix exists to remove. The kill-switch restores the + // previous fan-out faithfully: every peer, both requested planes. + const optimize = CATCHUP_STOP_ON_PROOF; + const needDurable = !optimize || !authorityProven.durable; const needSharedMemory = request.includeSharedMemory - && (!CATCHUP_STOP_ON_PROOF || !sharedMemoryProven()); + && (!optimize || !authorityProven.sharedMemory); + const fromAuthority = peerId === prepared.preferredPeerId; if (!needDurable) { const shared = needSharedMemory ? await runCatchupPlaneWithPolicy('foreground', syncSharedMemory) : null; - return { durable: null, shared }; + return { peerId, fromAuthority, durable: null, shared }; } - return runCatchupPlanesWithPolicy({ + const round = await runCatchupPlanesWithPolicy({ mode: 'foreground', includeSharedMemory: needSharedMemory, syncDurable, syncSharedMemory, }); + return { peerId, fromAuthority, ...round }; }; - const accumulate = ({ durable, shared }: PeerRound): void => { + const accumulate = ({ durable, shared, fromAuthority }: PeerRound): void => { let peerDenied = false; if (durable) { dataSynced += durable.insertedDataTriples ?? 0; @@ -240,6 +251,8 @@ async function runCatchup(request: CatchupRunRequest): Promise peerDenied = peerDenied || durable.deniedPhases > 0; if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { + const provenByData = (durable.insertedDataTriples ?? 0) > 0 + || durable.verifiedPrivateOnlyResponses > 0; if ((durable.insertedDataTriples ?? 0) > 0) { cleanPlaneCompletions.durable.verifiedDataPeers += 1; } @@ -249,6 +262,7 @@ async function runCatchup(request: CatchupRunRequest): Promise if ((durable.emptyResponses ?? 0) > 0) { cleanPlaneCompletions.durable.emptyPeers += 1; } + if (fromAuthority && provenByData) authorityProven.durable = true; } } @@ -276,6 +290,7 @@ async function runCatchup(request: CatchupRunRequest): Promise if (catchupPlaneCompletedWithoutFailure(shared)) { if ((shared.insertedDataTriples ?? 0) > 0) { cleanPlaneCompletions.sharedMemory.verifiedDataPeers += 1; + if (fromAuthority) authorityProven.sharedMemory = true; } if ((shared.emptyResponses ?? 0) > 0) { cleanPlaneCompletions.sharedMemory.emptyPeers += 1; @@ -354,9 +369,7 @@ async function runCatchup(request: CatchupRunRequest): Promise syncPeer, ); for (const round of rounds) accumulate(round); - const allRequestedPlanesProven = durableProven() - && (!request.includeSharedMemory || sharedMemoryProven()); - if (CATCHUP_STOP_ON_PROOF && allRequestedPlanesProven) break; + if (CATCHUP_STOP_ON_PROOF && authorityProvedEverything()) break; } diagnostics.noProtocolPeers = noProtocolPeers; diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index ccc6caed72..9ec2e71b2d 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -260,6 +260,36 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.dataSynced).toBe(peerIds.length); }); + it('keeps walking past a non-authoritative peer that returned verified data', async () => { + // A peer's `complete` flag proves it served ITS OWN manifest, not the union + // of what the network holds: peer-a can cleanly return KA-1 while peer-b + // holds KA-2 for the same graph. Without a resolved curator there is no + // reference snapshot, so a clean data-bearing round must NOT cut the walk + // short and strand peer-b's Knowledge Asset. + const peerIds = ['peer-a', 'peer-b', 'peer-c']; + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-disjoint', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls.sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.dataSynced).toBe(peerIds.length); + }); + it('opens at the full concurrency cap when no curator resolved', async () => { // A single-peer opening wave buys "one payload from the curator". Without a // resolvable curator it buys nothing, so the walk must not serialise the diff --git a/packages/cli/test/catchup-runner-worker-killswitch.test.ts b/packages/cli/test/catchup-runner-worker-killswitch.test.ts new file mode 100644 index 0000000000..07ff65395a --- /dev/null +++ b/packages/cli/test/catchup-runner-worker-killswitch.test.ts @@ -0,0 +1,164 @@ +// Pins the operator kill-switch for the issue #2006 progressive walk. +// +// `DKG_CATCHUP_STOP_ON_PROOF=0` must restore the PREVIOUS behaviour exactly: +// every sync-capable peer contacted, both requested planes pulled from each of +// them, and no early stop — so an operator can back the optimisation out +// without a redeploy if a graph ever lands short. The switch is read once at +// module load, which is why this lives in its own file. +import { describe, expect, it, vi } from 'vitest'; +import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS } from '@origintrail-official/dkg-agent'; +import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; + +vi.hoisted(() => { + process.env.DKG_CATCHUP_STOP_ON_PROOF = '0'; +}); + +const fakeParentPort = vi.hoisted(() => { + const messageListeners: Array<(message: any) => void> = []; + const port = { + on(event: string, listener: (message: any) => void) { + if (event === 'message') messageListeners.push(listener); + }, + onPosted: undefined as ((message: any) => void) | undefined, + postMessage(message: any) { + port.onPosted?.(message); + }, + emitMessage(message: any) { + for (const listener of messageListeners) listener(message); + }, + }; + return port; +}); + +vi.mock('node:worker_threads', async (importOriginal) => ({ + ...(await importOriginal()), + parentPort: fakeParentPort, +})); + +const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +function durableResult() { + return { + insertedTriples: 1, + complete: true, + fetchedMetaTriples: 0, + fetchedDataTriples: 1, + insertedMetaTriples: 0, + insertedDataTriples: 1, + bytesReceived: 10, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 1, + checkpointAdvances: 0, + emptyResponses: 0, + metaOnlyResponses: 0, + dataRejectedMissingMeta: 0, + rejectedKcs: 0, + failedPeers: 0, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + +function sharedResult() { + return { + insertedTriples: 1, + fetchedMetaTriples: 0, + fetchedDataTriples: 1, + insertedMetaTriples: 0, + insertedDataTriples: 1, + bytesReceived: 10, + resumedPhases: 0, + timedOutPhases: 0, + completedPhases: 1, + checkpointAdvances: 0, + emptyResponses: 0, + droppedDataTriples: 0, + failedPeers: 0, + failedPhases: 0, + deniedPhases: 0, + deferredBackpressure: 0, + }; +} + +let nextRunId = 1; + +async function runWorkerCatchup( + request: CatchupRunRequest, + handler: (method: string, args: unknown[]) => Promise, +): Promise { + await import('../src/catchup-runner-worker-impl.js'); + const runId = nextRunId++; + return new Promise((resolve, reject) => { + fakeParentPort.onPosted = (message: any) => { + if (message.type === 'invoke') { + handler(message.method, message.args).then( + (result) => fakeParentPort.emitMessage({ type: 'invoke-result', invokeId: message.invokeId, result }), + (error: unknown) => fakeParentPort.emitMessage({ + type: 'invoke-result', + invokeId: message.invokeId, + error: error instanceof Error ? error.message : String(error), + }), + ); + return; + } + if (message.type === 'run-result' && message.runId === runId) { + if (message.error) reject(new Error(message.error)); + else resolve(message.result as CatchupJobResult); + } + }; + fakeParentPort.emitMessage({ type: 'run', runId, request }); + }); +} + +describe('catch-up progressive walk kill-switch', () => { + it('restores the full fan-out over every peer and every requested plane', async () => { + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + let inFlight = 0; + let peak = 0; + + // The curator is first AND cleanly proves both planes on the very first + // peer — with the switch ON this run would stop after `peer-0`. + const result = await runWorkerCatchup({ contextGraphId: 'cg-killswitch', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(2); + inFlight -= 1; + return durableResult(); + } + case 'syncSharedMemory': { + sharedCalls.push(args[0] as string); + inFlight += 1; + peak = Math.max(peak, inFlight); + await delay(2); + inFlight -= 1; + return sharedResult(); + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(peerIds); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersTried).toBe(peerIds.length); + expect(result.peersNotAttempted).toBe(0); + expect(result.dataSynced).toBe(peerIds.length); + expect(result.sharedMemorySynced).toBe(peerIds.length); + // The pre-existing sync-storm bound still applies with the switch off. + expect(peak).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(peak).toBeGreaterThan(1); + }); +}); diff --git a/packages/cli/vitest.unit.config.ts b/packages/cli/vitest.unit.config.ts index cdca0b4175..04464f9d5e 100644 --- a/packages/cli/vitest.unit.config.ts +++ b/packages/cli/vitest.unit.config.ts @@ -68,6 +68,7 @@ export default defineConfig({ 'test/catchup-runner.test.ts', 'test/catchup-runner-worker-impl.test.ts', 'test/catchup-runner-worker-lifecycle.test.ts', + 'test/catchup-runner-worker-killswitch.test.ts', 'test/relay-status-block.test.ts', 'test/supervisor-liveness.test.ts', 'test/promote-async-routes.test.ts', From 4c02f87e528385bf357a5d092fa4f8ed543002d1 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 03:33:50 +0200 Subject: [PATCH 04/44] docs(sync): state where the admission-source trust boundary is Review round 1 follow-up: the `source?: string` fields are `string` rather than `SyncAdmissionSource` because they can be reconstructed from a postMessage payload across the catch-up Worker RPC boundary. Say so at each of the three option surfaces, and name the single clamp point. Co-Authored-By: Claude Opus 5 (1M context) --- packages/agent/src/dkg-agent-lifecycle.ts | 15 +++++++++++++-- packages/agent/src/sync/backpressure.ts | 8 +++++++- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 048517c592..0966f81318 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -943,7 +943,14 @@ export type DurableSyncOptions = { /** * Which trigger asked for this sync. Recorded as a bounded dimension on * node-wide scheduler diagnostics so queue pressure can be attributed to an - * origin; clamped to the closed `SyncAdmissionSource` set before use. + * origin. + * + * Typed `string`, not `SyncAdmissionSource`, on purpose: these options are + * reconstructed from a `postMessage` payload after crossing the catch-up + * Worker RPC boundary, where the compile-time union guarantees nothing. This + * is the untrusted edge; `normalizeSyncAdmissionSource` clamps it to the + * closed set once, in `acquire`, and every layer past that clamp is typed + * `SyncAdmissionSource`. */ source?: string; }; @@ -5235,7 +5242,11 @@ export class LifecycleSyncMethods extends DKGAgentBase { sharedMemorySyncPlan?: SharedMemorySyncContextGraphPlan; /** Admission override for foreground catch-up. */ priority?: number; - /** Bounded admission origin for node-wide scheduler diagnostics. */ + /** + * Bounded admission origin for node-wide scheduler diagnostics. `string` + * because it can arrive across the catch-up Worker RPC boundary; clamped + * to the closed `SyncAdmissionSource` set in `acquire`. + */ source?: string; }, ): Promise { diff --git a/packages/agent/src/sync/backpressure.ts b/packages/agent/src/sync/backpressure.ts index 6feb97d362..953be5414a 100644 --- a/packages/agent/src/sync/backpressure.ts +++ b/packages/agent/src/sync/backpressure.ts @@ -273,7 +273,13 @@ export async function withGlobalSyncBackpressure( lane?: SyncSchedulerLane; priority?: number; priorityClass?: SyncPriorityClass; - /** Which trigger enqueued this admission; clamped to the closed set. */ + /** + * Which trigger enqueued this admission. This is the untrusted edge of the + * label space — the value can originate across the catch-up Worker RPC + * boundary — so it is `string` here and clamped exactly once, below, by + * `normalizeSyncAdmissionSource`. Everything from the queue payload onward + * is typed `SyncAdmissionSource`. + */ source?: string; signal?: AbortSignal; /** Deterministic scheduler injection; not operator configuration. */ From 4e6ae9c531b32efbdb06439ecfa57a116943644f Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 03:36:24 +0200 Subject: [PATCH 05/44] docs(sync): correct the walk comment after authority gating The block still said the walk stops "as soon as every requested plane is proven by real verified data" and that it stops early even without a curator. Both became false when the stop was gated on authority proof. Co-Authored-By: Claude Opus 5 (1M context) --- .../cli/src/catchup-runner-worker-impl.ts | 25 +++++++++++-------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index dd7b3943ca..3bb5a7d330 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -324,27 +324,30 @@ async function runCatchup(request: CatchupRunRequest): Promise // node-wide sync-global queue (2 inflight / 4 queued) saturated against // itself. // - // Instead, walk escalating waves (1, 2, 4, ...) and stop as soon as every - // requested plane is proven by real verified data. Wave 1 is the curator when - // one is resolvable, so the happy path transfers exactly one payload. + // Instead, walk escalating waves and stop as soon as the AUTHORITY has proven + // every requested plane with verified data — see `authorityProven` above for + // why only the curator's snapshot may cut the walk short. Wave 1 is that + // curator when one is resolvable, so the happy path transfers exactly one + // payload; with no resolvable curator nothing is ever authority-proven and + // this degrades to the previous full bounded fan-out. // - // The stop condition is deliberately POSITIVE-only: an empty round proves + // The stop condition is also deliberately POSITIVE-only: an empty round proves // nothing on its own (an unrelated peer and an empty host are byte-identical // on the wire), so emptiness stays a whole-round verdict evaluated by // `catchupPlaneProvenByUnanimousEmpty` after every peer has been walked. // - // Tradeoff, stated deliberately: a peer's `complete` flag proves it served its - // own manifest, not that the manifest was network-complete. Foreground - // catch-up therefore optimises for one fast authoritative payload; breadth and - // eventual convergence remain the background reconcile lane's job. + // Tradeoff, stated deliberately: even the curator's `complete` flag proves it + // served its own manifest, not that the manifest was network-complete. + // Foreground catch-up therefore optimises for one fast authoritative payload; + // breadth and eventual convergence remain the background reconcile lane's job. // DKG_CATCHUP_STOP_ON_PROOF=0 restores the previous full fan-out. // // The single-peer opening wave is spent on the authority, so it is only taken // when there IS one: if no curator resolved (or it is not sync-capable), the // head of the ranked list has no special claim and serialising it would just - // add a round-trip to the front of every round. In that case the walk opens at - // the full concurrency cap — the previous first-round latency — and still - // stops as soon as something is proven. + // add a round-trip to the front of every round, with no early stop to earn it + // back. In that case the walk opens at the full concurrency cap — the previous + // first-round latency. const authorityFirst = prepared.preferredPeerId !== undefined && syncCapable[0] === prepared.preferredPeerId; const waveSizes = CATCHUP_STOP_ON_PROOF From def08c3a509b00afc47a922b71786fe1801a26fb Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 04:05:45 +0200 Subject: [PATCH 06/44] refactor(sync): type the catch-up plane boundary and normalize source once (review round 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * The skipped-plane model was built on `any`, so the accumulation block was the only statement of which fields each plane must carry. `PeerRound` now uses `CatchupDurableResult` / `CatchupSharedMemoryResult` derived from the agent's own `DurableSyncResult` / `SharedMemorySyncResult`, and the RPC helpers are typed rather than `invoke`. `null` stays the single exceptional case: a plane the authority already settled. * Admission source was threaded through the lifecycle layer as a raw string all the way to the scheduler, so a typo travelled as if valid and only collapsed at the final label. It is now normalized once, at the boundary where an untrusted value can enter (`runContextGraphSyncWithBackpressure`), and every layer past it carries `SyncAdmissionSource`. The clamp in `acquire` stays as defence in depth for anything reaching the scheduler by another route — the cast test pins that it still holds. * Rewrote the worker-lifecycle test against a fake `Worker` instead of spawning a real thread: a nested worker does not boot reliably inside vitest's multi-file pool (green alone, hung in a lane), and the contract under test is this file's handler wiring, not Node's documented 'exit'-on-terminate. Adds a double-close case and a 500ms race so a regression fails fast instead of burning the test timeout. Verified fail-before by neutering the handler: both cases fail. Co-Authored-By: Claude Opus 5 (1M context) --- packages/agent/src/dkg-agent-lifecycle.ts | 14 ++- packages/agent/src/index.ts | 2 + packages/agent/src/sync/backpressure.ts | 12 +- packages/agent/test/sync-backpressure.test.ts | 4 +- .../catchup-runner-worker-lifecycle.test.ts | 113 ++++++++++++++---- 5 files changed, 108 insertions(+), 37 deletions(-) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 0966f81318..69c815bffa 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -313,6 +313,7 @@ import { import { contextGraphPriority, countSyncPriorityClasses, + normalizeSyncAdmissionSource, orderContextGraphIdsByPriority, syncPriorityClass, type SyncAdmissionSource, @@ -1170,11 +1171,16 @@ export class LifecycleSyncMethods extends DKGAgentBase { /** Admission override for foreground catch-up / VM recovery. */ priorityOverride?: number; operationSignal?: AbortSignal; - /** Which trigger enqueued this admission; clamped to the closed set. */ + /** + * Which trigger enqueued this admission. Accepted as a loose string + * because it can arrive from the catch-up Worker RPC; normalized to the + * closed set HERE so every layer past this boundary carries the union. + */ source?: string; } = {}, ): Promise { - const { priorityOverride, operationSignal, source } = admission; + const { priorityOverride, operationSignal } = admission; + const source = normalizeSyncAdmissionSource(admission.source); const priority = priorityOverride ?? contextGraphPriority(this.config.syncContextGraphPriorities, contextGraphId); const admissionBoundary = combineSyncAdmissionSignals( @@ -4385,7 +4391,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds, onAccessDenied, options?.priority, - options?.source, + normalizeSyncAdmissionSource(options?.source), ); changelogResult = lane.result; legacyContextGraphIds = lane.remainingLegacyCgs; @@ -4765,7 +4771,7 @@ export class LifecycleSyncMethods extends DKGAgentBase { contextGraphIds: string[], onAccessDenied?: (contextGraphId: string) => void, priority?: number, - source?: string, + source?: SyncAdmissionSource, ): Promise<{ result?: DurableSyncResult; remainingLegacyCgs: string[] }> { const peerProtocols = await this.getPeerProtocols(remotePeerId); if (!peerProtocols.includes(PROTOCOL_SYNC_CHANGELOG)) { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index ccab71c045..a36f42e44c 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -225,6 +225,8 @@ export { type ImportedArtifactByteStore, type DurableSyncDiagnostics, type DurableSyncResult, + type SharedMemorySyncDiagnostics, + type SharedMemorySyncResult, } from './dkg-agent-types.js'; export { computeImportedArtifactSelector, diff --git a/packages/agent/src/sync/backpressure.ts b/packages/agent/src/sync/backpressure.ts index 953be5414a..7b4eb8d7aa 100644 --- a/packages/agent/src/sync/backpressure.ts +++ b/packages/agent/src/sync/backpressure.ts @@ -274,13 +274,13 @@ export async function withGlobalSyncBackpressure( priority?: number; priorityClass?: SyncPriorityClass; /** - * Which trigger enqueued this admission. This is the untrusted edge of the - * label space — the value can originate across the catch-up Worker RPC - * boundary — so it is `string` here and clamped exactly once, below, by - * `normalizeSyncAdmissionSource`. Everything from the queue payload onward - * is typed `SyncAdmissionSource`. + * Which trigger enqueued this admission. Callers normalize at the boundary + * where the value enters (`runContextGraphSyncWithBackpressure`); the clamp + * below is defence in depth for anything that reaches the scheduler by + * another route, so a bad cast can still only widen the label space to + * `unspecified`. */ - source?: string; + source?: SyncAdmissionSource; signal?: AbortSignal; /** Deterministic scheduler injection; not operator configuration. */ agingThresholdMs?: number; diff --git a/packages/agent/test/sync-backpressure.test.ts b/packages/agent/test/sync-backpressure.test.ts index f3e97b31e3..8e22e6b980 100644 --- a/packages/agent/test/sync-backpressure.test.ts +++ b/packages/agent/test/sync-backpressure.test.ts @@ -282,7 +282,9 @@ describe('sync global backpressure', () => { policy, ctx, label: 'durable:cg-x:peer-x', - source: 'leak-urn:cg:private:xyz', + // The option is typed `SyncAdmissionSource`; the cast is the point — + // the scheduler clamp is defence in depth for exactly this. + source: 'leak-urn:cg:private:xyz' as never, }, async () => new Promise((resolve) => { releaseRunning = resolve; diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index 4ea608ee30..f6a26cd343 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -1,38 +1,99 @@ // The daemon's subscribe job awaits `catchupRunner.run(...)` in a // fire-and-forget async IIFE with no timeout of its own. `close()` terminates -// the Worker, which emits 'exit' — never 'error' — so before this fix a pending -// run promise was simply never settled and the job stayed `running` with no -// `finishedAt` for the rest of the process's life. Issue #2006 makes that -// reachable routinely, because a walk can be in flight for much longer. -import { describe, expect, it } from 'vitest'; +// the Worker, and a terminated worker emits 'exit' — never 'error' — so before +// this fix a pending run promise was simply never settled and the job stayed +// `running` with no `finishedAt` for the rest of the process's life. Issue #2006 +// makes that reachable routinely, because a walk can be in flight much longer. +// +// `node:worker_threads` is mocked with a minimal fake Worker rather than +// spawning a real thread: nesting a real worker inside vitest's multi-file pool +// does not boot reliably, and the contract under test is this file's own +// handler wiring, not Node's (documented) 'exit'-on-terminate behaviour. +import { describe, expect, it, vi } from 'vitest'; import type { DKGAgent } from '@origintrail-official/dkg-agent'; -import { createCatchupRunner } from '../src/catchup-runner.js'; - -function hangingAgent(): DKGAgent { - return { - isPrivateContextGraph: async () => false, - resolvePreferredSyncPeerId: async () => undefined, - ensurePeerConnected: async () => {}, - // The worker's first RPC never settles, so the run stays in `pendingRuns`. - primeCatchupConnections: () => new Promise(() => {}), - selectCatchupPeers: (peers: unknown[]) => peers, - node: { libp2p: { getConnections: () => [] } }, - } as unknown as DKGAgent; + +type Listener = (...args: unknown[]) => void; + +const workerControl = vi.hoisted(() => { + const state = { + listeners: new Map(), + posted: [] as unknown[], + terminated: false, + }; + class FakeWorker { + constructor(_path: string) { + state.listeners.clear(); + state.posted.length = 0; + state.terminated = false; + } + + on(event: string, listener: Listener) { + const existing = state.listeners.get(event) ?? []; + existing.push(listener); + state.listeners.set(event, existing); + } + + postMessage(message: unknown) { + state.posted.push(message); + } + + async terminate() { + state.terminated = true; + // Node emits 'exit' for a terminated worker, with code 1. It does NOT + // emit 'error' — which is exactly why the 'exit' handler is needed. + for (const listener of state.listeners.get('exit') ?? []) listener(1); + return 1; + } + } + return { state, FakeWorker }; +}); + +vi.mock('node:worker_threads', async (importOriginal) => ({ + ...(await importOriginal()), + Worker: workerControl.FakeWorker, +})); + +const { createCatchupRunner } = await import('../src/catchup-runner.js'); + +const stubAgent = {} as unknown as DKGAgent; + +/** Fail fast on a regression: an unsettled run must not burn the test timeout. */ +function withinTick(promise: Promise): Promise { + return Promise.race([ + promise, + new Promise<'still-pending'>((resolve) => { setTimeout(() => resolve('still-pending'), 500); }), + ]); } describe('WorkerCatchupRunner lifecycle', () => { it('rejects an in-flight run when the worker exits instead of leaving it pending forever', async () => { - const runner = createCatchupRunner(hangingAgent()); + const runner = createCatchupRunner(stubAgent); const run = runner.run({ contextGraphId: 'cg-hang', includeSharedMemory: false }); - const settled = run.then(() => 'resolved' as const, () => 'rejected' as const); + const settled = run.then(() => 'resolved' as const, (error: Error) => error); + + // The run was dispatched and is awaiting a `run-result` that will never come. + expect(workerControl.state.posted).toHaveLength(1); + expect(workerControl.state.posted[0]).toMatchObject({ type: 'run' }); + + await runner.close(); + + const outcome = await withinTick(settled); + expect(outcome).toBeInstanceOf(Error); + expect((outcome as Error).message).toContain('exited'); + }); - // Give the worker a moment to boot and issue its first invoke. - await new Promise((resolve) => setTimeout(resolve, 200)); + it('rejects every pending run exactly once', async () => { + const runner = createCatchupRunner(stubAgent); + const first = runner.run({ contextGraphId: 'cg-a', includeSharedMemory: false }) + .then(() => 'resolved' as const, () => 'rejected' as const); + const second = runner.run({ contextGraphId: 'cg-b', includeSharedMemory: true }) + .then(() => 'resolved' as const, () => 'rejected' as const); + + await runner.close(); + // A second close (or a late 'error' after 'exit') must not double-settle. await runner.close(); - await expect(Promise.race([ - settled, - new Promise((resolve) => setTimeout(() => resolve('pending'), 5_000)), - ])).resolves.toBe('rejected'); - }, 30_000); + await expect(withinTick(Promise.all([first, second]))) + .resolves.toEqual(['rejected', 'rejected']); + }); }); From 216bcbe5a16ad4e0fb135c9635d36b76670183c8 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 04:28:34 +0200 Subject: [PATCH 07/44] fix(sync): make the empty verdict reachable, latch worker death, cover the authority gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent adversarial pass over the final diff found four real defects, three of them introduced by this PR. All four are fixed here. * The empty verdict was unreachable in practice. It required `fetchedMetaTriples === 0`, but EVERY registered Context Graph carries definition triples in its own `/_meta`, so any peer that hosts the graph returns metadata even when the graph holds zero Knowledge Assets — a legitimately empty public graph would have been permanently `unreachable` rather than merely unproven. It also voided on `failedPeers`, a transport failure to a peer we never heard from; on the live testnet run in this PR that was 6 of 9 peers. Both clauses are gone. What remains is what the issue actually asked for: void when data was fetched, or when a peer engaged and then failed (`failedPhases` / `timedOutPhases` / `deniedPhases` / `deferredBackpressure`). The reported #2006 shape is still killed twice over. * The curator answering cleanly EMPTY now settles a plane for the walk. Shared memory is routinely empty for a graph with durable data and `includeSharedMemory` defaults to true on subscribe, so keying the stop purely on inserted rows meant the headline optimisation would rarely fire in the shape it targets. * The worker `'exit'` handler settled in-flight runs but latched nothing. The runner is constructed once per daemon and `postMessage` to a dead worker neither throws nor delivers, so after a crash every LATER subscribe hung at `running` too — and the route's dedupe then handed that stuck job back on every retry. The failure is latched; later runs reject immediately. * The authority gate added in review round 1 had no executable coverage: every proving test either had the curator prove, or ran a single wave, so deleting `fromAuthority` from both sites failed nothing. Two tests now cover the negative direction (>4 peers, no curator, wave-1 peer proving), and removing either gate fails both. Also adds the durable-only early stop, the authority-clean-empty stop, and the `durable: null` skipped-plane round. * `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS=` (blank, the normal docker-compose/systemd shape for "unset") parsed as 0 and silently disabled all retries. Blank is now treated as unset; an explicit 0 still disables. * CHANGELOG entry and operator documentation for both new env knobs. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 18 ++ docs/use-dkg/backpressure-observability.md | 11 + packages/agent/src/sync/catchup-policy.ts | 10 +- .../agent/test/catchup-concurrency.test.ts | 5 + .../cli/src/catchup-runner-worker-impl.ts | 17 +- packages/cli/src/catchup-runner.ts | 58 +++-- .../test/catchup-runner-worker-impl.test.ts | 210 +++++++++++++++--- packages/cli/test/catchup-runner.test.ts | 34 ++- 8 files changed, 314 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e61008818..4b77b376c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,24 @@ All notable changes to the DKG V10 node are documented here. The format is based ## [Unreleased] +### Fixed + +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Metadata and unreachable peers are deliberately not treated as content, so a registered public graph that genuinely holds nothing still settles cleanly. +- **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s), and the timer is unreferenced so a pending backoff cannot outlive shutdown. +- **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. + +### Changed + +- **`sync-global` scheduler diagnostics attribute queue pressure to a trigger** (#2006): the `operation` dimension in `GET /api/diagnostics/backpressure` and in the `[backpressure]` log records changes from the work class alone (`durable`, which merely duplicated `lane`) to `:` — for example `durable:catchup-foreground` versus `durable:on-connect` or `durable:reconcile`. Both halves are closed sets, so the label space stays bounded and free of Context Graph and peer identifiers; an unrecognised source clamps to `unspecified`. Dashboards that group on `operation` for the `sync-global` scheduler will see the new values. `GET /api/sync/catchup-status` gains `result.peersNotAttempted`, the count of sync-capable peers the walk deliberately skipped. + +### Operator knobs + +| Variable | Default | Effect | +| --- | --- | --- | +| `DKG_CATCHUP_STOP_ON_PROOF` | on | Set to `0`/`false`/`no`/`off` to restore the pre-#2006 full fan-out: every sync-capable peer, both requested planes, no early stop. The escape hatch for the deliberate tradeoff that foreground catch-up may land the curator's snapshot rather than the union of every peer's. | +| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend waiting for local `sync-global` capacity. An explicit `0` disables retries; a blank value is treated as unset. | +| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Unchanged. Caps in-flight per-peer sync rounds and now also caps the widest escalation wave. | + ## [10.0.11] - 2026-07-30 A focused stability release on two paths a busy node exercises constantly. One external `/api/query` read could amplify into a planner-stalling query that starved every other subsystem: a caller that had already constrained `GRAPH ?g` to a handful of verified partitions was rewritten with a second `VALUES ?g` carrying the entire allow-list, expanding a 3 KB query to roughly 24 KB and occupying the store for minutes, cascading into queue-wait timeouts across promotion, gossip validation, SWM catch-up, and durable sync. External reads now run on the store scheduler's background lane, a disconnected caller's store work is cancelled instead of orphaned, and the redundant graph rewrite is elided. Separately, the `dkg integration` CLI is brought back into line with the registry's published JSON Schema, which its parser had drifted *stricter* than — so no `manual` entry was readable at all, in either the CLI or the node dashboard's integrations sidebar. The dashboard database stays at 31 — no migration. **No smart-contract changes — no deployment required** (no Solidity source, ABI, or deployment-registry changes since 10.0.10). diff --git a/docs/use-dkg/backpressure-observability.md b/docs/use-dkg/backpressure-observability.md index d79ffe5f2b..03ab0d895c 100644 --- a/docs/use-dkg/backpressure-observability.md +++ b/docs/use-dkg/backpressure-observability.md @@ -87,6 +87,17 @@ it — as `:`: | `swm-recovery` | curator-targeted shared-memory recovery | | `unspecified` | a caller that did not declare an origin | +### Tuning foreground catch-up + +Two knobs govern the foreground Context Graph catch-up that most often shows up +as `catchup-foreground` pressure. Both are read once at daemon start. + +| Variable | Default | Effect | +| --- | --- | --- | +| `DKG_CATCHUP_STOP_ON_PROOF` | on | The catch-up walks peers in escalating waves and stops once the resolved curator has settled every requested plane. Set to `0`, `false`, `no`, or `off` to restore the previous behaviour: every sync-capable peer, both requested planes, no early stop. Use this if a graph ever lands short — foreground catch-up optimises for one authoritative payload, while breadth remains the background reconcile lane's job. | +| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend waiting for local `sync-global` capacity before the job reports a retryable `deferred`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. | +| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Caps in-flight per-peer sync rounds, and therefore the widest escalation wave. Raising it above the `sync-global` queue depth lets a single catch-up saturate the scheduler against itself. | + So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}` in a `queuedOperations` summary reads as "four explicit catch-up durable admissions are queued, the oldest for 109 seconds", and the matching diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index dc5728987f..f0af789308 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -26,8 +26,14 @@ export const CATCHUP_BACKPRESSURE_JITTER_RATIO = 0.25; * pinning it at `running` forever. */ export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = (() => { - const raw = Number(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); - return Number.isInteger(raw) && raw >= 0 ? raw : 180_000; + // A blank env var is the normal docker-compose / `.env` / systemd shape for + // "not set", and `Number('')` is 0 — which would silently disable retries + // entirely, strictly worse than the ladder this replaced. Treat empty as + // unset; an explicit `0` still means "do not retry". + const raw = process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS?.trim(); + if (!raw) return 180_000; + const parsed = Number(raw); + return Number.isInteger(parsed) && parsed >= 0 ? parsed : 180_000; })(); /** Bounded admission origin recorded on node-wide scheduler diagnostics. */ diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts index dc5c749955..06c387fb7b 100644 --- a/packages/agent/test/catchup-concurrency.test.ts +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -51,7 +51,12 @@ describe('catchupWaveSizes', () => { }); it('keeps the shared fan-out cap a small positive number', () => { + // The cap is env-overridable, so this is a guard on operator input as much + // as on the default: a cap above the sync-global queue depth would let one + // catch-up saturate the scheduler against itself, which is the shape of the + // 2026-07-07 sync storm. expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeLessThanOrEqual(16); + expect(Number.isInteger(CATCHUP_MAX_CONCURRENT_PEER_SYNCS)).toBe(true); }); }); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 3bb5a7d330..8c092ba8dc 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -262,7 +262,13 @@ async function runCatchup(request: CatchupRunRequest): Promise if ((durable.emptyResponses ?? 0) > 0) { cleanPlaneCompletions.durable.emptyPeers += 1; } - if (fromAuthority && provenByData) authorityProven.durable = true; + // The curator answering cleanly settles this plane whether it carried + // data or was legitimately empty: "the host says there is nothing here" + // is the authoritative empty proof, and without it a graph with no + // public data on one plane could never stop the walk. + if (fromAuthority && (provenByData || (durable.emptyResponses ?? 0) > 0)) { + authorityProven.durable = true; + } } } @@ -290,11 +296,18 @@ async function runCatchup(request: CatchupRunRequest): Promise if (catchupPlaneCompletedWithoutFailure(shared)) { if ((shared.insertedDataTriples ?? 0) > 0) { cleanPlaneCompletions.sharedMemory.verifiedDataPeers += 1; - if (fromAuthority) authorityProven.sharedMemory = true; } if ((shared.emptyResponses ?? 0) > 0) { cleanPlaneCompletions.sharedMemory.emptyPeers += 1; } + // Same rule as durable: the curator settles the plane by answering + // cleanly, with data or empty. Shared memory is frequently empty for a + // graph that has durable data, and `includeSharedMemory` defaults to + // true on subscribe, so without this the early stop would almost never + // fire in the shape the fix targets. + if (fromAuthority && ((shared.insertedDataTriples ?? 0) > 0 || (shared.emptyResponses ?? 0) > 0)) { + authorityProven.sharedMemory = true; + } } } diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 199416af10..6189b7033b 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -554,12 +554,29 @@ export function catchupPlaneProvenByData( * empty (`sync-verify-worker-impl.ts`), so an empty response can never carry * hosting evidence — there is no per-peer signal that could distinguish the two. * - * Emptiness is therefore only provable as a verdict over the whole round: - * at least one peer completed cleanly empty, nobody delivered any content, and - * nothing failed, timed out, was denied, or was deferred. One data-bearing peer - * that fails is enough to void it — that exact shape (122,705 triples fetched, - * five failed phases, five unrelated peers answering empty) is what settled - * issue #2006's run as `done` with 1 KA out of 40. + * Emptiness is therefore a verdict over the whole round: some peer completed + * cleanly empty, nobody delivered any graph CONTENT, and no peer engaged and + * then failed part-way. That exact shape — 122,705 data triples fetched and + * five failed phases, with five unrelated peers answering empty — is what + * settled issue #2006's run as `done` with 1 KA out of 40, and either clause + * kills it on its own. + * + * Two counters are deliberately NOT consulted: + * + * - `fetchedMetaTriples`. Every registered Context Graph carries definition + * triples in its own `/_meta`, so any peer that hosts the graph at all + * returns metadata even when the graph holds zero Knowledge Assets. Treating + * metadata as content would make a legitimately empty public graph + * permanently unreadable rather than merely unproven. + * - `failedPeers`. That is a transport failure to a peer we never heard from — + * on a live testnet a majority of connected peers can be unreachable — and an + * unreachable stranger is evidence of nothing. A peer that DID engage and + * then failed shows up in `failedPhases` / `timedOutPhases` / `deniedPhases` + * / `deferredBackpressure`, all of which do void the verdict. + * + * Residual, unchanged from before this rule existed: if the only host is + * unreachable while another peer answers cleanly empty, the round still reads + * as empty. Readiness is re-derived on the next catch-up. */ export function catchupPlaneProvenByUnanimousEmpty( completion: CatchupPlaneCompletionEvidence | undefined, @@ -574,9 +591,7 @@ export function catchupPlaneProvenByUnanimousEmpty( || (diagnostics?.emptyResponses ?? 0) > 0; if (!cleanEmptyObserved) return false; if ((diagnostics?.fetchedDataTriples ?? 0) > 0) return false; - if ((diagnostics?.fetchedMetaTriples ?? 0) > 0) return false; - return (diagnostics?.failedPeers ?? 0) === 0 - && (diagnostics?.failedPhases ?? 0) === 0 + return (diagnostics?.failedPhases ?? 0) === 0 && (diagnostics?.timedOutPhases ?? 0) === 0 && (diagnostics?.deniedPhases ?? 0) === 0 && (diagnostics?.deferredBackpressure ?? 0) === 0; @@ -692,6 +707,8 @@ class WorkerCatchupRunner implements CatchupRunner { private readonly worker: Worker; private nextRunId = 0; private readonly pendingRuns = new Map(); + /** Set once the worker dies; every later run fails fast instead of hanging. */ + private workerFailure: Error | undefined; constructor(private readonly agent: DKGAgent) { const jsWorkerUrl = new URL('./catchup-runner-worker-impl.js', import.meta.url); @@ -712,26 +729,31 @@ class WorkerCatchupRunner implements CatchupRunner { } }); this.worker.on('error', (error) => { - this.rejectPendingRuns(error); + this.fail(error); }); - // `close()` terminates the worker, which emits 'exit' — never 'error'. - // Without this handler every in-flight `run()` promise stays pending - // forever, so the daemon's fire-and-forget subscribe job is pinned at - // `running` with no `finishedAt` for the rest of the process's life. + // `close()` terminates the worker, which emits 'exit' — never 'error'. A + // crashed or terminated worker is also permanent: the runner is constructed + // once per daemon and `postMessage` to a dead worker neither throws nor + // delivers. Without this, an in-flight run stayed pending forever AND every + // later run did too, so the daemon's fire-and-forget subscribe jobs were + // pinned at `running` with no `finishedAt` for the rest of the process's + // life — and the route's dedupe then hands that stuck job back on every + // re-subscribe, so an operator cannot even retrigger. this.worker.on('exit', (code) => { - this.rejectPendingRuns( - new Error(`Catch-up worker exited (code ${code}) before the run completed`), - ); + this.fail(new Error(`Catch-up worker exited (code ${code}) before the run completed`)); }); } - private rejectPendingRuns(error: Error): void { + /** Latch the terminal failure and settle everything waiting on the worker. */ + private fail(error: Error): void { + this.workerFailure ??= error; const pending = [...this.pendingRuns.values()]; this.pendingRuns.clear(); for (const run of pending) run.reject(error); } run(request: CatchupRunRequest): Promise { + if (this.workerFailure) return Promise.reject(this.workerFailure); const runId = this.nextRunId++; return new Promise((resolve, reject) => { this.pendingRuns.set(runId, { resolve, reject }); diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 9ec2e71b2d..143da35a51 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -9,10 +9,11 @@ // probes) may ever be in flight, per-peer failures stay isolated, and the // aggregation keeps its one-result-per-peer input-order shape; // * the issue #2006 progressive walk: peers are contacted in escalating -// waves over the authority-ranked list and the walk STOPS as soon as every -// requested plane is proven by verified data, so the happy path transfers -// one payload instead of one per peer. An empty response proves nothing on -// its own and can never stop the walk. +// waves over the authority-ranked list and the walk STOPS as soon as the +// RESOLVED CURATOR has settled every requested plane, so the happy path +// transfers one payload instead of one per peer. A non-authoritative peer's +// clean round settles nothing — it can neither stop the walk nor narrow a +// later peer to one plane. import { describe, expect, it, vi } from 'vitest'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, @@ -21,8 +22,8 @@ import { } from '@origintrail-official/dkg-agent'; import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; -// The foreground backpressure budget is wall-clock (default 60s). Shrink it for -// this file so the persistently-deferred case settles quickly; the exact +// The foreground backpressure budget is wall-clock (default 180s). Shrink it +// for this file so the persistently-deferred case settles quickly; the exact // deadline arithmetic is pinned deterministically in // packages/agent/test/catchup-policy.test.ts with an injected clock. vi.hoisted(() => { @@ -262,11 +263,15 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) it('keeps walking past a non-authoritative peer that returned verified data', async () => { // A peer's `complete` flag proves it served ITS OWN manifest, not the union - // of what the network holds: peer-a can cleanly return KA-1 while peer-b - // holds KA-2 for the same graph. Without a resolved curator there is no + // of what the network holds: peer-0 can cleanly return KA-1 while a later + // peer holds KA-2 for the same graph. Without a resolved curator there is no // reference snapshot, so a clean data-bearing round must NOT cut the walk - // short and strand peer-b's Knowledge Asset. - const peerIds = ['peer-a', 'peer-b', 'peer-c']; + // short and strand the other peers' Knowledge Assets. + // + // The peer set must exceed the concurrency cap, or the whole walk is one + // wave and this assertion could not fail regardless of the gate. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); const durableCalls: string[] = []; const result = await runWorkerCatchup({ contextGraphId: 'cg-disjoint', includeSharedMemory: false }, async (method, args) => { @@ -285,11 +290,170 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) } }); - expect(durableCalls.sort()).toEqual([...peerIds].sort()); + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); expect(result.peersNotAttempted).toBe(0); expect(result.dataSynced).toBe(peerIds.length); }); + it('does not let a non-authoritative peer narrow a later peer to one plane', async () => { + // The other half of the authority gate: a non-curator peer settling shared + // memory must not cause later peers to be contacted for durable only. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + await runWorkerCatchup({ contextGraphId: 'cg-no-narrow', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: undefined, isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { ...durableResult(), complete: false }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + }); + + it('stops after the curator proves the only requested plane', async () => { + // Plain `subscribe` with no workspace is the most common production shape + // for the early stop; the multi-wave curator case above requests both + // planes, so this pins the durable-only path. + const peerIds = Array.from({ length: 10 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-durable-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + }); + + it('lets the curator settle a plane by answering cleanly empty', async () => { + // Shared memory is frequently empty for a graph that has durable data, and + // `includeSharedMemory` defaults to true on subscribe. If only inserted + // rows could settle a plane, the early stop would almost never fire in the + // shape this fix targets. + const peerIds = Array.from({ length: 10 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-swm', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(sharedCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.sharedMemory.emptyPeers).toBe(1); + }); + + it('skips the durable plane on fallback peers once the curator settled it', async () => { + // The `durable: null` round — the reason the peer-accounting helpers accept + // a missing plane at all. The curator settles durable but not shared + // memory, so later peers must be contacted for shared memory only and must + // not be credited with a durable response. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-fallback', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + // The curator engages and fails (so SWM is never settled); every + // fallback peer transport-fails, delivering nothing at all. + return args[0] === 'peer-0' + ? { + ...sharedResult(), + insertedTriples: 0, + insertedDataTriples: 0, + completedPhases: 0, + failedPhases: 1, + } + : { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 0, + failedPeers: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Durable pulled once, from the curator; shared memory from everyone. + expect(durableCalls).toEqual(['peer-0']); + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersTried).toBe(peerIds.length); + expect(result.peersNotAttempted).toBe(0); + // The skipped durable plane must not manufacture a response for peers whose + // only requested plane transport-failed: only the curator responded. + expect(result.peersResponded).toBe(1); + expect(result.peersSucceeded).toBe(0); + // One durable round in the whole walk — that is the amplification fix. + expect(result.diagnostics?.durable.fetchedDataTriples).toBe(1); + expect(result.dataSynced).toBe(1); + }); + it('opens at the full concurrency cap when no curator resolved', async () => { // A single-peer opening wave buys "one payload from the curator". Without a // resolvable curator it buys nothing, so the walk must not serialise the @@ -354,12 +518,11 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); }); - it('narrows fallback peers to the planes still in question', async () => { - // A Context Graph whose public durable data is empty can never prove its - // durable plane by verified data, so the walk must cover every peer to - // establish the whole-round empty verdict. Without per-plane narrowing that - // would drag a full re-pull of the ALREADY PROVEN shared-memory plane out of - // every remaining peer — the exact amplification this fix removes. + it('narrows fallback peers to the planes the curator already settled', async () => { + // The curator settles shared memory but never settles durable (its durable + // round engages and fails). Without per-plane narrowing, walking on for + // durable would drag a full re-pull of the ALREADY SETTLED shared-memory + // plane out of every remaining peer — the exact amplification this removes. const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); const durableCalls: string[] = []; const sharedCalls: string[] = []; @@ -374,12 +537,11 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) durableCalls.push(args[0] as string); return { ...durableResult(), + complete: false, insertedTriples: 0, - fetchedDataTriples: 0, insertedDataTriples: 0, - bytesReceived: 0, - completedPhases: 2, - emptyResponses: 1, + completedPhases: 0, + timedOutPhases: 1, }; case 'syncSharedMemory': sharedCalls.push(args[0] as string); @@ -391,14 +553,12 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) } }); - // Shared memory is proven by the first peer and never pulled again… + // Shared memory is settled by the curator and never pulled again… expect(sharedCalls).toEqual(['peer-0']); - // …while the (cheap, empty) durable plane still walks everyone, because a - // clean empty answer only proves emptiness as a whole-round verdict. + // …while the unsettled durable plane still walks everyone. expect(durableCalls).toEqual(peerIds); expect(result.sharedMemorySynced).toBe(1); expect(result.cleanPlaneCompletions?.sharedMemory.verifiedDataPeers).toBe(1); - expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(peerIds.length); }); it('keeps per-peer failure isolation and probe filtering under the bounded fan-out', async () => { diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index b86e70d90b..5da5ee00c9 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -683,8 +683,6 @@ describe('catch-up plane proof predicates', () => { it.each([ ['a data-bearing peer that failed', { fetchedDataTriples: 122_705, failedPhases: 5 }], ['fetched data with no verified completion', { fetchedDataTriples: 5_000 }], - ['a peer that returned metadata', { fetchedMetaTriples: 12 }], - ['a transport failure', { failedPeers: 1 }], ['a failed phase', { failedPhases: 1 }], ['a timed-out phase', { timedOutPhases: 1 }], ['a denial', { deniedPhases: 1 }], @@ -696,6 +694,38 @@ describe('catch-up plane proof predicates', () => { expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(false); }); + it.each([ + // Every registered Context Graph carries definition triples in its own + // `/_meta`, so ANY peer that hosts the graph returns metadata even when + // the graph holds zero Knowledge Assets. Treating that as content would make + // a legitimately empty public graph permanently unreadable. + ['metadata from a hosting peer', { fetchedMetaTriples: 12 }], + // A transport failure is a peer we never heard from. On a live testnet a + // majority of connected peers can be unreachable; an unreachable stranger is + // evidence of nothing. A peer that DID engage and then failed shows up in + // the voiding counters above. + ['a transport failure to an unreachable peer', { failedPeers: 4 }], + ])('still proves an empty public round despite %s', (_label, overrides) => { + const diagnostics = { ...cleanEmptyRound, ...overrides }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(true); + }); + + it('proves a registered public graph that simply has no Knowledge Assets yet', () => { + // The shape a freshly registered, still-empty public Context Graph actually + // produces: its host serves the CG definition triples (metadata) and no + // data, other peers answer clean-empty, and some connected peers are + // unreachable. This must reach `done`, not sit at `unreachable` forever. + const registeredButEmpty = { + ...cleanEmptyRound, + fetchedMetaTriples: 9, + emptyResponses: 3, + failedPeers: 2, + }; + expect(catchupPlaneReady(emptyPeers, registeredButEmpty, { isPrivate: false })).toBe(true); + }); + it('still reports ready when a peer delivered verified data despite other failures', () => { const diagnostics = { ...cleanEmptyRound, fetchedDataTriples: 24_541, failedPhases: 1 }; const completion = { ...emptyPeers, verifiedDataPeers: 1 }; From 598c0ca56f4ec325e53a779de41ed6650bb5326d Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 04:34:39 +0200 Subject: [PATCH 08/44] test(sync): cover the legacy readiness branch and both empty-evidence carriers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Coverage gaps the adversarial pass flagged as nits: - the rewritten no-`cleanPlaneCompletions` compatibility branch had no test, so a rolling-upgrade result could have bypassed the round-level guard unnoticed; now pinned in both directions - every fixture set both `completion.emptyPeers` and `diagnostics.emptyResponses`, so neither disjunct was individually pinned — which matters because the legacy branch can only supply the aggregate one - a wave-size assertion carried a comment describing a different test Co-Authored-By: Claude Opus 5 (1M context) --- .../test/catchup-runner-worker-impl.test.ts | 6 ++-- packages/cli/test/catchup-runner.test.ts | 18 +++++++++++ .../context-graph-catchup-readiness.test.ts | 32 +++++++++++++++++++ 3 files changed, 53 insertions(+), 3 deletions(-) diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 143da35a51..f32c106040 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -18,7 +18,6 @@ import { describe, expect, it, vi } from 'vitest'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, FOREGROUND_CATCHUP_SYNC_PRIORITY, - catchupWaveSizes, } from '@origintrail-official/dkg-agent'; import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; @@ -250,8 +249,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(peakProbes).toBeLessThanOrEqual(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); // …but later waves are still actually parallel, not accidentally serialised. expect(peakSyncs).toBeGreaterThan(1); - // The first wave is a single peer, so a proving authority costs one payload. - expect(catchupWaveSizes(peerIds.length, CATCHUP_MAX_CONCURRENT_PEER_SYNCS)[0]).toBe(1); + // No curator resolved here, so the opening wave is NOT narrowed to one peer + // (that narrowing only buys anything when there is an authority to spend it + // on). The ranked order is still honoured. expect(startOrder[0]).toBe('peer-0'); // Coverage preserved when nothing proves: every peer walked, in rank order. diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 5da5ee00c9..0189378ad5 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -740,6 +740,24 @@ describe('catch-up plane proof predicates', () => { { isPrivate: false }, )).toBe(false); }); + + it('accepts either evidence carrier for the clean empty completion', () => { + // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter + // (`diagnostics.emptyResponses`) are separate carriers, and the legacy + // no-`cleanPlaneCompletions` branch in the readiness classifier can only + // supply the aggregate one. Pin each independently so neither disjunct can + // be dropped unnoticed. + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 1 }, + { ...cleanEmptyRound, emptyResponses: 0 }, + { isPrivate: false }, + )).toBe(true); + expect(catchupPlaneProvenByUnanimousEmpty( + noEvidence, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, + )).toBe(true); + }); }); describe('catch-up peer accounting with a skipped plane', () => { diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 61f10a507b..64edd4a5b0 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -277,6 +277,38 @@ describe('context graph catch-up readiness classification', () => { expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); }); + it('applies the same fail-closed empty rule to a legacy runner result', () => { + // A result without `cleanPlaneCompletions` (an older in-process runner + // during a rolling upgrade) takes the compatibility branch. It must not be + // a way around the round-level guard. + const masked = publicEmptyRoundResult(); + delete masked.cleanPlaneCompletions; + masked.diagnostics!.durable.fetchedDataTriples = 122_705; + masked.diagnostics!.durable.failedPhases = 5; + + expect(classifyContextGraphCatchupReadiness({ + result: masked, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }).jobStatus).not.toBe('done'); + + // …and a legacy result from a genuinely empty round still settles. + const clean = publicEmptyRoundResult(); + delete clean.cleanPlaneCompletions; + expect(classifyContextGraphCatchupReadiness({ + result: clean, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + readinessPatch: { durableVerified: true }, + }); + }); + it('never proves a private plane from an empty round', () => { const classification = classifyContextGraphCatchupReadiness({ result: publicEmptyRoundResult(), From 1930228bacb6b1b891dfcc6e7f2107a49d58cf0a Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 05:34:20 +0200 Subject: [PATCH 09/44] test(sync): pin the reconciler's admission origin at its call site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI (Tornado: agent [2/10]) caught a real call-shape break: `trySyncFromPeer` gained a third argument — the bounded admission origin — and `sync-on-connect-churn` asserted the exact argument list. Updated to assert 'reconcile', which is the point of the change: reconciler queue pressure must be attributable rather than indistinguishable from sync-on-connect. Co-Authored-By: Claude Opus 5 (1M context) --- .../agent/test/sync-on-connect-churn.test.ts | 5 +- .../deployments/localhost_contracts.json | 192 +++++++++--------- 2 files changed, 100 insertions(+), 97 deletions(-) diff --git a/packages/agent/test/sync-on-connect-churn.test.ts b/packages/agent/test/sync-on-connect-churn.test.ts index 92803e51fd..fc786c95df 100644 --- a/packages/agent/test/sync-on-connect-churn.test.ts +++ b/packages/agent/test/sync-on-connect-churn.test.ts @@ -132,7 +132,10 @@ describe('sync-on-connect churn gates', () => { await (agent as any).reconcileSyncFromConnectedPeers(); await flushTimers(); - expect(trySyncFromPeer.calls).toEqual([[PEER_A, expect.any(Function)]]); + // The third argument is the bounded admission origin (issue #2006): the + // reconciler's queue pressure must be attributable to `reconcile`, not + // indistinguishable from sync-on-connect. + expect(trySyncFromPeer.calls).toEqual([[PEER_A, expect.any(Function), 'reconcile']]); }); it('records backoff after a failed sync round and blocks connection-open rescheduling', async () => { diff --git a/packages/evm-module/deployments/localhost_contracts.json b/packages/evm-module/deployments/localhost_contracts.json index 0e83e1f9f1..4b5623a386 100644 --- a/packages/evm-module/deployments/localhost_contracts.json +++ b/packages/evm-module/deployments/localhost_contracts.json @@ -3,289 +3,289 @@ "Hub": { "evmAddress": "0x5FbDB2315678afecb367f032d93F642f64180aa3", "version": "1.0.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 1, - "deploymentTimestamp": 1784536683273, + "deploymentTimestamp": 1785555112901, "deployed": true }, "Token": { "evmAddress": "0xe7f1725E7734CE288F8367e1Bb143E90bb3F0512", "version": null, - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 2, - "deploymentTimestamp": 1784536683462, + "deploymentTimestamp": 1785555114173, "deployed": true }, "ParametersStorage": { "evmAddress": "0xe70f935c32dA4dB13e7876795f1e175465e6458e", "version": "10.0.4", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 205, - "deploymentTimestamp": 1784536683996, + "deploymentTimestamp": 1785555117556, "deployed": true }, "WhitelistStorage": { "evmAddress": "0x2625760C4A8e8101801D3a48eE64B2bEA42f1E96", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 211, - "deploymentTimestamp": 1784536684356, + "deploymentTimestamp": 1785555119542, "deployed": true }, "IdentityStorage": { "evmAddress": "0xD6b040736e948621c5b6E0a494473c47a6113eA8", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 213, - "deploymentTimestamp": 1784536684604, + "deploymentTimestamp": 1785555120997, "deployed": true }, "ShardingTableStorage": { "evmAddress": "0xAdE429ba898c34722e722415D722A70a297cE3a2", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 215, - "deploymentTimestamp": 1784536684812, + "deploymentTimestamp": 1785555122244, "deployed": true }, "StakingStorage": { "evmAddress": "0xcE0066b1008237625dDDBE4a751827de037E53D2", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 217, - "deploymentTimestamp": 1784536685052, + "deploymentTimestamp": 1785555123538, "deployed": true }, "ProfileStorage": { "evmAddress": "0x51C65cd0Cdb1A8A8b79dfc2eE965B1bA0bb8fc89", "version": "10.0.4", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 220, - "deploymentTimestamp": 1784536685312, + "deploymentTimestamp": 1785555125095, "deployed": true }, "Chronos": { "evmAddress": "0xC7143d5bA86553C06f5730c8dC9f8187a621A8D4", "version": null, - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 222, - "deploymentTimestamp": 1784536685498, + "deploymentTimestamp": 1785555126178, "deployed": true }, "EpochStorageV8": { "evmAddress": "0xc9952Fc93Fa9bE383ccB39008c786b9f94eAc95d", "version": "10.0.4", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 224, - "deploymentTimestamp": 1784536685716, + "deploymentTimestamp": 1785555127317, "deployed": true }, "DKGKnowledgeAssets": { "evmAddress": "0x70eE76691Bdd9696552AF8d4fd634b3cF79DD529", "version": "10.1.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 227, - "deploymentTimestamp": 1784536685969, + "deploymentTimestamp": 1785555128701, "deployed": true }, "AskStorage": { "evmAddress": "0x162700d1613DfEC978032A909DE02643bC55df1A", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 230, - "deploymentTimestamp": 1784536686178, + "deploymentTimestamp": 1785555129853, "deployed": true }, "Identity": { "evmAddress": "0xcD0048A5628B37B8f743cC2FeA18817A29e97270", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 233, - "deploymentTimestamp": 1784536686390, + "deploymentTimestamp": 1785555130974, "deployed": true }, "ConvictionStakingStorage": { "evmAddress": "0x942ED2fa862887Dc698682cc6a86355324F0f01e", "version": "10.0.6", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 236, - "deploymentTimestamp": 1784536686641, + "deploymentTimestamp": 1785555132223, "deployed": true }, "ShardingTable": { "evmAddress": "0xa722bdA6968F50778B973Ae2701e90200C564B49", "version": "10.0.3", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 239, - "deploymentTimestamp": 1784536686857, + "deploymentTimestamp": 1785555133427, "deployed": true }, "Ask": { "evmAddress": "0xe1708FA6bb2844D5384613ef0846F9Bc1e8eC55E", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 242, - "deploymentTimestamp": 1784536687068, + "deploymentTimestamp": 1785555134734, "deployed": true }, "RandomSamplingStorage": { "evmAddress": "0x871ACbEabBaf8Bed65c22ba7132beCFaBf8c27B5", "version": "10.2.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 245, - "deploymentTimestamp": 1784536687297, + "deploymentTimestamp": 1785555136127, "deployed": true }, "StakingKPI": { "evmAddress": "0x683d9CDD3239E0e01E8dC6315fA50AD92aB71D2d", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 248, - "deploymentTimestamp": 1784536687516, + "deploymentTimestamp": 1785555137252, "deployed": true }, "Profile": { "evmAddress": "0x71a0b8A2245A9770A4D887cE1E4eCc6C1d4FF28c", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 251, - "deploymentTimestamp": 1784536687749, + "deploymentTimestamp": 1785555138317, "deployed": true }, "ContextGraphStorage": { "evmAddress": "0x193521C8934bCF3473453AF4321911E7A89E0E12", "version": "10.0.6", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 254, - "deploymentTimestamp": 1784536687967, + "deploymentTimestamp": 1785555139475, "deployed": true }, "ContextGraphValueStorage": { "evmAddress": "0x3C1Cb427D20F15563aDa8C249E71db76d7183B6c", "version": "10.0.2", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 257, - "deploymentTimestamp": 1784536688181, + "deploymentTimestamp": 1785555140659, "deployed": true }, "CGWeightTreeStorage": { "evmAddress": "0x547382C0D1b23f707918D3c83A77317B71Aa8470", "version": "1.0.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 260, - "deploymentTimestamp": 1784536688408, + "deploymentTimestamp": 1785555141701, "deployed": true }, "RandomSampling": { "evmAddress": "0x5e6CB7E728E1C320855587E1D9C6F7972ebdD6D5", "version": "10.6.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 263, - "deploymentTimestamp": 1784536688652, + "deploymentTimestamp": 1785555145659, "deployed": true }, "ContextGraphWaiverStorage": { "evmAddress": "0xeAd789bd8Ce8b9E94F5D0FCa99F8787c7e758817", "version": "1.0.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 266, - "deploymentTimestamp": 1784536688860, + "deploymentTimestamp": 1785555147116, "deployed": true }, "ContextGraphs": { "evmAddress": "0xd9fEc8238711935D6c8d79Bef2B9546ef23FC046", "version": "10.0.4", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 268, - "deploymentTimestamp": 1784536689076, + "deploymentTimestamp": 1785555148673, "deployed": true }, "PublishingConvictionStorage": { "evmAddress": "0x9fD16eA9E31233279975D99D5e8Fc91dd214c7Da", "version": "10.0.3", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 271, - "deploymentTimestamp": 1784536689315, + "deploymentTimestamp": 1785555150576, "deployed": true }, "PublishingConviction": { "evmAddress": "0xb932C8342106776E73E39D695F3FFC3A9624eCE0", "version": "10.0.8", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 274, - "deploymentTimestamp": 1784536689532, + "deploymentTimestamp": 1785555152888, "deployed": true }, "DKGPublishingConvictionNFT": { "evmAddress": "0x2c8ED11fd7A058096F2e5828799c68BE88744E2F", "version": "10.0.3", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 277, - "deploymentTimestamp": 1784536689751, + "deploymentTimestamp": 1785555155123, "deployed": true }, "KnowledgeAssetsLifecycle": { "evmAddress": "0x572316aC11CB4bc5daf6BDae68f43EA3CCE3aE0e", "version": "10.1.6", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 280, - "deploymentTimestamp": 1784536690002, + "deploymentTimestamp": 1785555157436, "deployed": true }, "StakingV10": { "evmAddress": "0xCd7c00Ac6dc51e8dCc773971Ac9221cC582F3b1b", "version": "10.0.5", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 283, - "deploymentTimestamp": 1784536690227, + "deploymentTimestamp": 1785555165637, "deployed": true }, "DKGStakingConvictionNFT": { "evmAddress": "0xCa1D199b6F53Af7387ac543Af8e8a34455BBe5E0", "version": "10.0.3", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 286, - "deploymentTimestamp": 1784536690455, + "deploymentTimestamp": 1785555168788, "deployed": true }, "MigrationCreditRecovery": { "evmAddress": "0xFD2Cf3b56a73c75A7535fFe44EBABe7723c64719", "version": "1.0.0", - "gitBranch": "fix/swm-catchup-materialize", - "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", + "gitBranch": "fix/2006-catchup-peer-selection", + "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", "deploymentBlock": 289, - "deploymentTimestamp": 1784536690699, + "deploymentTimestamp": 1785555170846, "deployed": true } } From 96a48d2b9e6da7a1bded47364626f3cd97fb2fc1 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 05:35:12 +0200 Subject: [PATCH 10/44] chore: drop the local hardhat deployment artifact from this PR `packages/evm-module/deployments/localhost_contracts.json` is regenerated by local EVM test runs and was swept into the previous commit by `git add -A`. Restored to the base revision so the PR diff carries no unrelated churn. Co-Authored-By: Claude Opus 5 (1M context) --- .../deployments/localhost_contracts.json | 192 +++++++++--------- 1 file changed, 96 insertions(+), 96 deletions(-) diff --git a/packages/evm-module/deployments/localhost_contracts.json b/packages/evm-module/deployments/localhost_contracts.json index 4b5623a386..0e83e1f9f1 100644 --- a/packages/evm-module/deployments/localhost_contracts.json +++ b/packages/evm-module/deployments/localhost_contracts.json @@ -3,289 +3,289 @@ "Hub": { "evmAddress": "0x5FbDB2315678afecb367f032d93F642f64180aa3", "version": "1.0.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 1, - "deploymentTimestamp": 1785555112901, + "deploymentTimestamp": 1784536683273, "deployed": true }, "Token": { "evmAddress": "0xe7f1725E7734CE288F8367e1Bb143E90bb3F0512", "version": null, - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 2, - "deploymentTimestamp": 1785555114173, + "deploymentTimestamp": 1784536683462, "deployed": true }, "ParametersStorage": { "evmAddress": "0xe70f935c32dA4dB13e7876795f1e175465e6458e", "version": "10.0.4", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 205, - "deploymentTimestamp": 1785555117556, + "deploymentTimestamp": 1784536683996, "deployed": true }, "WhitelistStorage": { "evmAddress": "0x2625760C4A8e8101801D3a48eE64B2bEA42f1E96", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 211, - "deploymentTimestamp": 1785555119542, + "deploymentTimestamp": 1784536684356, "deployed": true }, "IdentityStorage": { "evmAddress": "0xD6b040736e948621c5b6E0a494473c47a6113eA8", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 213, - "deploymentTimestamp": 1785555120997, + "deploymentTimestamp": 1784536684604, "deployed": true }, "ShardingTableStorage": { "evmAddress": "0xAdE429ba898c34722e722415D722A70a297cE3a2", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 215, - "deploymentTimestamp": 1785555122244, + "deploymentTimestamp": 1784536684812, "deployed": true }, "StakingStorage": { "evmAddress": "0xcE0066b1008237625dDDBE4a751827de037E53D2", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 217, - "deploymentTimestamp": 1785555123538, + "deploymentTimestamp": 1784536685052, "deployed": true }, "ProfileStorage": { "evmAddress": "0x51C65cd0Cdb1A8A8b79dfc2eE965B1bA0bb8fc89", "version": "10.0.4", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 220, - "deploymentTimestamp": 1785555125095, + "deploymentTimestamp": 1784536685312, "deployed": true }, "Chronos": { "evmAddress": "0xC7143d5bA86553C06f5730c8dC9f8187a621A8D4", "version": null, - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 222, - "deploymentTimestamp": 1785555126178, + "deploymentTimestamp": 1784536685498, "deployed": true }, "EpochStorageV8": { "evmAddress": "0xc9952Fc93Fa9bE383ccB39008c786b9f94eAc95d", "version": "10.0.4", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 224, - "deploymentTimestamp": 1785555127317, + "deploymentTimestamp": 1784536685716, "deployed": true }, "DKGKnowledgeAssets": { "evmAddress": "0x70eE76691Bdd9696552AF8d4fd634b3cF79DD529", "version": "10.1.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 227, - "deploymentTimestamp": 1785555128701, + "deploymentTimestamp": 1784536685969, "deployed": true }, "AskStorage": { "evmAddress": "0x162700d1613DfEC978032A909DE02643bC55df1A", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 230, - "deploymentTimestamp": 1785555129853, + "deploymentTimestamp": 1784536686178, "deployed": true }, "Identity": { "evmAddress": "0xcD0048A5628B37B8f743cC2FeA18817A29e97270", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 233, - "deploymentTimestamp": 1785555130974, + "deploymentTimestamp": 1784536686390, "deployed": true }, "ConvictionStakingStorage": { "evmAddress": "0x942ED2fa862887Dc698682cc6a86355324F0f01e", "version": "10.0.6", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 236, - "deploymentTimestamp": 1785555132223, + "deploymentTimestamp": 1784536686641, "deployed": true }, "ShardingTable": { "evmAddress": "0xa722bdA6968F50778B973Ae2701e90200C564B49", "version": "10.0.3", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 239, - "deploymentTimestamp": 1785555133427, + "deploymentTimestamp": 1784536686857, "deployed": true }, "Ask": { "evmAddress": "0xe1708FA6bb2844D5384613ef0846F9Bc1e8eC55E", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 242, - "deploymentTimestamp": 1785555134734, + "deploymentTimestamp": 1784536687068, "deployed": true }, "RandomSamplingStorage": { "evmAddress": "0x871ACbEabBaf8Bed65c22ba7132beCFaBf8c27B5", "version": "10.2.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 245, - "deploymentTimestamp": 1785555136127, + "deploymentTimestamp": 1784536687297, "deployed": true }, "StakingKPI": { "evmAddress": "0x683d9CDD3239E0e01E8dC6315fA50AD92aB71D2d", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 248, - "deploymentTimestamp": 1785555137252, + "deploymentTimestamp": 1784536687516, "deployed": true }, "Profile": { "evmAddress": "0x71a0b8A2245A9770A4D887cE1E4eCc6C1d4FF28c", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 251, - "deploymentTimestamp": 1785555138317, + "deploymentTimestamp": 1784536687749, "deployed": true }, "ContextGraphStorage": { "evmAddress": "0x193521C8934bCF3473453AF4321911E7A89E0E12", "version": "10.0.6", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 254, - "deploymentTimestamp": 1785555139475, + "deploymentTimestamp": 1784536687967, "deployed": true }, "ContextGraphValueStorage": { "evmAddress": "0x3C1Cb427D20F15563aDa8C249E71db76d7183B6c", "version": "10.0.2", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 257, - "deploymentTimestamp": 1785555140659, + "deploymentTimestamp": 1784536688181, "deployed": true }, "CGWeightTreeStorage": { "evmAddress": "0x547382C0D1b23f707918D3c83A77317B71Aa8470", "version": "1.0.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 260, - "deploymentTimestamp": 1785555141701, + "deploymentTimestamp": 1784536688408, "deployed": true }, "RandomSampling": { "evmAddress": "0x5e6CB7E728E1C320855587E1D9C6F7972ebdD6D5", "version": "10.6.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 263, - "deploymentTimestamp": 1785555145659, + "deploymentTimestamp": 1784536688652, "deployed": true }, "ContextGraphWaiverStorage": { "evmAddress": "0xeAd789bd8Ce8b9E94F5D0FCa99F8787c7e758817", "version": "1.0.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 266, - "deploymentTimestamp": 1785555147116, + "deploymentTimestamp": 1784536688860, "deployed": true }, "ContextGraphs": { "evmAddress": "0xd9fEc8238711935D6c8d79Bef2B9546ef23FC046", "version": "10.0.4", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 268, - "deploymentTimestamp": 1785555148673, + "deploymentTimestamp": 1784536689076, "deployed": true }, "PublishingConvictionStorage": { "evmAddress": "0x9fD16eA9E31233279975D99D5e8Fc91dd214c7Da", "version": "10.0.3", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 271, - "deploymentTimestamp": 1785555150576, + "deploymentTimestamp": 1784536689315, "deployed": true }, "PublishingConviction": { "evmAddress": "0xb932C8342106776E73E39D695F3FFC3A9624eCE0", "version": "10.0.8", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 274, - "deploymentTimestamp": 1785555152888, + "deploymentTimestamp": 1784536689532, "deployed": true }, "DKGPublishingConvictionNFT": { "evmAddress": "0x2c8ED11fd7A058096F2e5828799c68BE88744E2F", "version": "10.0.3", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 277, - "deploymentTimestamp": 1785555155123, + "deploymentTimestamp": 1784536689751, "deployed": true }, "KnowledgeAssetsLifecycle": { "evmAddress": "0x572316aC11CB4bc5daf6BDae68f43EA3CCE3aE0e", "version": "10.1.6", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 280, - "deploymentTimestamp": 1785555157436, + "deploymentTimestamp": 1784536690002, "deployed": true }, "StakingV10": { "evmAddress": "0xCd7c00Ac6dc51e8dCc773971Ac9221cC582F3b1b", "version": "10.0.5", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 283, - "deploymentTimestamp": 1785555165637, + "deploymentTimestamp": 1784536690227, "deployed": true }, "DKGStakingConvictionNFT": { "evmAddress": "0xCa1D199b6F53Af7387ac543Af8e8a34455BBe5E0", "version": "10.0.3", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 286, - "deploymentTimestamp": 1785555168788, + "deploymentTimestamp": 1784536690455, "deployed": true }, "MigrationCreditRecovery": { "evmAddress": "0xFD2Cf3b56a73c75A7535fFe44EBABe7723c64719", "version": "1.0.0", - "gitBranch": "fix/2006-catchup-peer-selection", - "gitCommitHash": "598c0ca56f4ec325e53a779de41ed6650bb5326d", + "gitBranch": "fix/swm-catchup-materialize", + "gitCommitHash": "02e52c8caa44bc5b72017d9f6b8faf25e308b7a7", "deploymentBlock": 289, - "deploymentTimestamp": 1785555170846, + "deploymentTimestamp": 1784536690699, "deployed": true } } From 687650fda1cd34ef5f3d167e989eddd6a4d8d724 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 06:38:34 +0200 Subject: [PATCH 11/44] refactor(sync): one definition of "a peer's plane evidence" (review round 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five follow-up review replies; three applied, two answered in thread. * The worker hand-coded the plane-proof rule at the early-stop boundary, duplicating `catchupPlaneProvenByData`'s semantics for a single peer. A future verified-content signal added to the readiness predicate would have left the walk's stop condition silently behind. `catchupPeerPlaneEvidence` now reduces ONE peer's round to the evidence a round accumulates, and both sides go through the same predicate — the walk applies it to one peer's evidence, readiness to the round's sum. Mutation-checked: dropping the verified-private-only signal from the reducer fails the worker suite. * The shared-memory-only fallback branch runs through a different call path than the both-planes one, so it has to carry foreground admission itself. Now asserted: every fallback peer's `syncSharedMemory` receives FOREGROUND_CATCHUP_SYNC_PRIORITY and `catchup-foreground`. * The in-agent foreground runner could have dropped the admission source while keeping the priority, and the existing test would still have passed — inline foreground catch-up would then report as `durable:unspecified` in node-wide diagnostics. The coalescing test now records and asserts the source alongside the priority. Co-Authored-By: Claude Opus 5 (1M context) --- .../agent/test/sync-fetch-coalescing.test.ts | 16 ++++- .../cli/src/catchup-runner-worker-impl.ts | 67 +++++++++---------- packages/cli/src/catchup-runner.ts | 37 ++++++++++ .../test/catchup-runner-worker-impl.test.ts | 10 +++ 4 files changed, 93 insertions(+), 37 deletions(-) diff --git a/packages/agent/test/sync-fetch-coalescing.test.ts b/packages/agent/test/sync-fetch-coalescing.test.ts index e9071f98af..beb7a8840b 100644 --- a/packages/agent/test/sync-fetch-coalescing.test.ts +++ b/packages/agent/test/sync-fetch-coalescing.test.ts @@ -790,6 +790,7 @@ describe('DKGAgent sync fetch coalescing', () => { const remotePeer = { toString: () => PEER_A }; const order: string[] = []; const priorities: Array = []; + const sources: Array = []; let durableCalls = 0; try { @@ -802,10 +803,11 @@ describe('DKGAgent sync fetch coalescing', () => { _onPhase: unknown, _onAccessDenied: unknown, _sinceBatchIdFor: unknown, - options: { priority?: number } | undefined, + options: { priority?: number; source?: string } | undefined, ) => { durableCalls += 1; priorities.push(options?.priority); + sources.push(options?.source); order.push(`durable-${durableCalls}`); return durableCalls === 1 ? { @@ -818,9 +820,10 @@ describe('DKGAgent sync fetch coalescing', () => { (agent as any).syncSharedMemoryFromPeerDetailed = async ( _peerId: string, _contextGraphIds: string[], - options: { priority?: number } | undefined, + options: { priority?: number; source?: string } | undefined, ) => { priorities.push(options?.priority); + sources.push(options?.source); order.push('shared'); return cleanSharedMemorySyncResult(); }; @@ -839,6 +842,15 @@ describe('DKGAgent sync fetch coalescing', () => { FOREGROUND_CATCHUP_SYNC_PRIORITY, FOREGROUND_CATCHUP_SYNC_PRIORITY, ]); + // The admission ORIGIN travels with the priority on the in-agent runner + // too: dropping it here while keeping the priority would silently report + // inline foreground catch-up as `durable:unspecified` in node-wide + // scheduler diagnostics (issue #2006). + expect(sources).toEqual([ + 'catchup-foreground', + 'catchup-foreground', + 'catchup-foreground', + ]); } finally { await agent.stop().catch(() => {}); } diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 8c092ba8dc..8c31cea4ed 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -9,9 +9,11 @@ import { runCatchupPlanesWithPolicy, } from '@origintrail-official/dkg-agent'; import { + addCatchupPlaneEvidence, + catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, - catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByData, type CatchupJobResult, type CatchupRunRequest, } from './catchup-runner.js'; @@ -250,25 +252,21 @@ async function runCatchup(request: CatchupRunRequest): Promise (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); peerDenied = peerDenied || durable.deniedPhases > 0; - if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { - const provenByData = (durable.insertedDataTriples ?? 0) > 0 - || durable.verifiedPrivateOnlyResponses > 0; - if ((durable.insertedDataTriples ?? 0) > 0) { - cleanPlaneCompletions.durable.verifiedDataPeers += 1; - } - if (durable.verifiedPrivateOnlyResponses > 0) { - cleanPlaneCompletions.durable.verifiedPrivateOnlyPeers += 1; - } - if ((durable.emptyResponses ?? 0) > 0) { - cleanPlaneCompletions.durable.emptyPeers += 1; - } - // The curator answering cleanly settles this plane whether it carried - // data or was legitimately empty: "the host says there is nothing here" - // is the authoritative empty proof, and without it a graph with no - // public data on one plane could never stop the walk. - if (fromAuthority && (provenByData || (durable.emptyResponses ?? 0) > 0)) { - authorityProven.durable = true; - } + const durableEvidence = catchupPeerPlaneEvidence(durable, { complete: durable.complete }); + addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); + // The curator answering cleanly settles this plane whether it carried + // data or was legitimately empty: "the host says there is nothing here" + // is the authoritative empty proof, and without it a graph with no + // public data on one plane could never stop the walk. + // + // The positive half runs through the SAME predicate the readiness + // classifier uses, just applied to one peer's evidence rather than the + // round's, so the stop condition cannot drift from the readiness rule. + if ( + fromAuthority + && (catchupPlaneProvenByData(durableEvidence) || durableEvidence.emptyPeers > 0) + ) { + authorityProven.durable = true; } } @@ -293,21 +291,20 @@ async function runCatchup(request: CatchupRunRequest): Promise (diagnostics.sharedMemory.deniedPhases ?? 0) + (shared.deniedPhases ?? 0); peerDenied = peerDenied || shared.deniedPhases > 0; - if (catchupPlaneCompletedWithoutFailure(shared)) { - if ((shared.insertedDataTriples ?? 0) > 0) { - cleanPlaneCompletions.sharedMemory.verifiedDataPeers += 1; - } - if ((shared.emptyResponses ?? 0) > 0) { - cleanPlaneCompletions.sharedMemory.emptyPeers += 1; - } - // Same rule as durable: the curator settles the plane by answering - // cleanly, with data or empty. Shared memory is frequently empty for a - // graph that has durable data, and `includeSharedMemory` defaults to - // true on subscribe, so without this the early stop would almost never - // fire in the shape the fix targets. - if (fromAuthority && ((shared.insertedDataTriples ?? 0) > 0 || (shared.emptyResponses ?? 0) > 0)) { - authorityProven.sharedMemory = true; - } + // Shared memory carries no verified-private-only signal, so the shared + // evidence only ever has data/empty set — the same reducer still applies. + const sharedEvidence = catchupPeerPlaneEvidence(shared); + addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); + // Same rule as durable: the curator settles the plane by answering + // cleanly, with data or empty. Shared memory is frequently empty for a + // graph that has durable data, and `includeSharedMemory` defaults to + // true on subscribe, so without this the early stop would almost never + // fire in the shape the fix targets. + if ( + fromAuthority + && (catchupPlaneProvenByData(sharedEvidence) || sharedEvidence.emptyPeers > 0) + ) { + authorityProven.sharedMemory = true; } } diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 6189b7033b..48845862f2 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -531,6 +531,43 @@ export interface CatchupPlaneRoundDiagnostics { deferredBackpressure?: number; } +/** + * Reduce ONE peer's plane result to the evidence a round accumulates from it. + * + * This is the single definition of what a peer's round contributes, so the + * walk's stop condition and the readiness classifier cannot drift: the walk + * feeds one peer's evidence to {@link catchupPlaneProvenByData}, and readiness + * feeds the summed evidence to the same predicate. Adding a new verified-content + * signal therefore has exactly one place to change. + * + * A plane that did not complete cleanly contributes nothing at all. + */ +export function catchupPeerPlaneEvidence( + plane: (CatchupPhaseProgress & { emptyResponses?: number }) | null | undefined, + options: { complete?: boolean } = {}, +): CatchupPlaneCompletionEvidence { + const none = { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }; + if (!plane || !catchupPlaneCompletedWithoutFailure(plane, options.complete)) return none; + return { + verifiedDataPeers: (plane.insertedDataTriples ?? 0) > 0 ? 1 : 0, + verifiedPrivateOnlyPeers: (plane.verifiedPrivateOnlyResponses ?? 0) > 0 ? 1 : 0, + emptyPeers: (plane.emptyResponses ?? 0) > 0 ? 1 : 0, + }; +} + +/** Fold one peer's evidence into the running per-plane totals. */ +export function addCatchupPlaneEvidence( + total: CatchupPlaneCompletionEvidence, + peer: CatchupPlaneCompletionEvidence, +): void { + total.verifiedDataPeers += peer.verifiedDataPeers; + if (peer.verifiedPrivateOnlyPeers) { + total.verifiedPrivateOnlyPeers = (total.verifiedPrivateOnlyPeers ?? 0) + + peer.verifiedPrivateOnlyPeers; + } + total.emptyPeers += peer.emptyPeers; +} + /** * Positive proof: some peer cleanly completed this plane while carrying * cryptographically verified content. This is the only evidence strong enough diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index f32c106040..47cfd9803c 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -402,6 +402,8 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); const durableCalls: string[] = []; const sharedCalls: string[] = []; + const sharedPriorities: Array = []; + const sharedSources: Array = []; const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-fallback', includeSharedMemory: true }, async (method, args) => { switch (method) { @@ -414,6 +416,8 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) return durableResult(); case 'syncSharedMemory': sharedCalls.push(args[0] as string); + sharedPriorities.push(args[2] as number | undefined); + sharedSources.push(args[3] as string | undefined); // The curator engages and fails (so SWM is never settled); every // fallback peer transport-fails, delivering nothing at all. return args[0] === 'peer-0' @@ -443,6 +447,12 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) // Durable pulled once, from the curator; shared memory from everyone. expect(durableCalls).toEqual(['peer-0']); expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + // The shared-only fallback goes through a different call path than the + // both-planes one, so it has to carry foreground admission itself. + expect(sharedPriorities).toEqual( + peerIds.map(() => FOREGROUND_CATCHUP_SYNC_PRIORITY), + ); + expect(sharedSources).toEqual(peerIds.map(() => 'catchup-foreground')); expect(result.peersTried).toBe(peerIds.length); expect(result.peersNotAttempted).toBe(0); // The skipped durable plane must not manufacture a response for peers whose From 04f1216dab7ac89960c55ce84747863eaa3bd2a4 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 10:05:02 +0200 Subject: [PATCH 12/44] fix(sync): a bootstrap hint must not count as the catch-up authority (review round 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweeping the review threads properly (paginated) surfaced six I had never seen, including two blockers. * 🔴 The authority marker was `prepared.preferredPeerId`, which `resolvePreferredSyncPeerId` falls back to from the authenticated join-approval hint when metadata resolves no curator. That hint can be stale — a curator that has since rotated its libp2p identity leaves an ordinary member on that peer id — so it could stop the walk exactly as if it were the curator, which is the risk the authority gate exists to close. `resolveAuthoritativeSyncPeerId` returns the preferred peer ONLY when it is no longer the live bootstrap hint (metadata resolution deletes the hint), and the walk keys `fromAuthority` and the opening-wave narrowing off that. The hint still orders the walk; it just cannot end it. * 🔴 The clean-empty regression test could not observe early stopping: every peer sat in the first wave, so a regression accepting any clean-empty round as proof would still have contacted all of them. The empty peers now fill the whole first wave and the data-bearing peer sits behind a wave boundary. Mutation-checked: dropping `fromAuthority` from the empty branch now fails it, and two sibling tests. * A run started AFTER the worker died had no coverage — only runs already pending at exit. The latch could have been deleted silently. * The cap test asserted `<= 16`, a bound production does not enforce, so a valid `DKG_CATCHUP_MAX_CONCURRENT_PEERS=32` would have failed it. It now asserts the actual contract. * The blank-vs-zero env contract had no test, because the constant resolves at module load. Extracted `resolveCatchupBackpressureMaxWaitMs` and pinned all four documented inputs; mutation-checked. * `defaultWait`'s `unref()` was never observed — every other test injects `wait`. Now asserted via a `setTimeout` spy; mutation-checked. * CHANGELOG documents the removed `CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` export and `retryDelaysMs` option, so the removal is declared rather than silent, with the migration path. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 4 ++ packages/agent/src/dkg-agent-lifecycle.ts | 27 ++++++++ packages/agent/src/index.ts | 2 + packages/agent/src/sync/catchup-policy.ts | 34 +++++++--- .../agent/test/catchup-concurrency.test.ts | 15 +++-- packages/agent/test/catchup-policy.test.ts | 67 +++++++++++++++++++ .../cli/src/catchup-runner-worker-impl.ts | 13 +++- packages/cli/src/catchup-runner.ts | 8 +++ .../test/catchup-runner-worker-impl.test.ts | 65 +++++++++++++++--- .../catchup-runner-worker-killswitch.test.ts | 2 +- .../catchup-runner-worker-lifecycle.test.ts | 18 +++++ 11 files changed, 226 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b77b376c5..ac47341f7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,10 @@ All notable changes to the DKG V10 node are documented here. The format is based - **`sync-global` scheduler diagnostics attribute queue pressure to a trigger** (#2006): the `operation` dimension in `GET /api/diagnostics/backpressure` and in the `[backpressure]` log records changes from the work class alone (`durable`, which merely duplicated `lane`) to `:` — for example `durable:catchup-foreground` versus `durable:on-connect` or `durable:reconcile`. Both halves are closed sets, so the label space stays bounded and free of Context Graph and peer identifiers; an unrecognised source clamps to `unspecified`. Dashboards that group on `operation` for the `sync-global` scheduler will see the new values. `GET /api/sync/catchup-status` gains `result.peersNotAttempted`, the count of sync-capable peers the walk deliberately skipped. +### Removed + +- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. + ### Operator knobs | Variable | Default | Effect | diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 69c815bffa..729ff443ed 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -6307,6 +6307,33 @@ export class LifecycleSyncMethods extends DKGAgentBase { return curatorPeerId ?? this.preferredSyncPeers.get(contextGraphId); } + /** + * The preferred sync peer, but only when it came from authoritative Context + * Graph metadata rather than the bootstrap join-approval hint. + * + * `resolveCuratorPeerId` deletes the hint the moment metadata resolves a + * curator, so a returned peer that is STILL the live hint means metadata did + * not resolve one and we are looking at a bootstrap value that can be stale + * (a curator that has since rotated its libp2p identity, leaving an ordinary + * member on that peer id). + * + * Callers that merely want to try the best peer first should keep using + * {@link resolvePreferredSyncPeerId}. Only callers that let one peer's answer + * stand for the whole graph — the foreground catch-up walk's early stop — + * need this stricter notion, because a non-curator that happens to be ranked + * first must never be able to cut the walk short. + */ + async resolveAuthoritativeSyncPeerId( + this: DKGAgent, + contextGraphId: string, + ): Promise { + const preferredPeerId = await this.resolvePreferredSyncPeerId(contextGraphId); + if (!preferredPeerId) return undefined; + return preferredPeerId === this.preferredSyncPeers.get(contextGraphId) + ? undefined + : preferredPeerId; + } + async ensurePeerConnected(this: DKGAgent, peerId: string): Promise { await ensurePeerConnectedAtom(this.node.libp2p as any, this.discovery, peerId); if (await this.networkAdmissionCoordinator.ensureAdmitted(peerId, createOperationContext('connect'))) return; diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index a36f42e44c..496905ab49 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -319,6 +319,8 @@ export { CATCHUP_BACKPRESSURE_JITTER_RATIO, CATCHUP_BACKPRESSURE_MAX_DELAY_MS, CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + resolveCatchupBackpressureMaxWaitMs, FOREGROUND_CATCHUP_SYNC_PRIORITY, catchupPriorityForMode, catchupSourceForMode, diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index f0af789308..342711432b 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -25,16 +25,30 @@ export const CATCHUP_BACKPRESSURE_JITTER_RATIO = 0.25; * saturated node fails the catch-up job with a retryable status instead of * pinning it at `running` forever. */ -export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = (() => { - // A blank env var is the normal docker-compose / `.env` / systemd shape for - // "not set", and `Number('')` is 0 — which would silently disable retries - // entirely, strictly worse than the ladder this replaced. Treat empty as - // unset; an explicit `0` still means "do not retry". - const raw = process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS?.trim(); - if (!raw) return 180_000; - const parsed = Number(raw); - return Number.isInteger(parsed) && parsed >= 0 ? parsed : 180_000; -})(); +export const DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = 180_000; + +/** + * Parse the operator-facing retry budget. + * + * Exported as a pure function because the constant below is resolved once at + * module load, which makes the env contract untestable in place — and the + * contract has a sharp edge worth pinning: a BLANK assignment is the normal + * docker-compose / `.env` / systemd shape for "not set", but `Number('')` is + * `0`, which would silently disable retries entirely and land strictly worse + * than the fixed ladder this replaced. Blank is unset; an explicit `0` still + * means "do not retry". + */ +export function resolveCatchupBackpressureMaxWaitMs(raw: string | undefined): number { + const trimmed = raw?.trim(); + if (!trimmed) return DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + const parsed = Number(trimmed); + return Number.isInteger(parsed) && parsed >= 0 + ? parsed + : DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; +} + +export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = + resolveCatchupBackpressureMaxWaitMs(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); /** Bounded admission origin recorded on node-wide scheduler diagnostics. */ export type CatchupAdmissionSource = 'catchup-foreground' | 'catchup-background'; diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts index 06c387fb7b..2cf583f429 100644 --- a/packages/agent/test/catchup-concurrency.test.ts +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -50,13 +50,14 @@ describe('catchupWaveSizes', () => { expect(catchupWaveSizes(-2, 4)).toEqual([]); }); - it('keeps the shared fan-out cap a small positive number', () => { - // The cap is env-overridable, so this is a guard on operator input as much - // as on the default: a cap above the sync-global queue depth would let one - // catch-up saturate the scheduler against itself, which is the shape of the - // 2026-07-07 sync storm. - expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); - expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeLessThanOrEqual(16); + it('resolves the shared fan-out cap to a positive integer', () => { + // Deliberately NOT asserting an upper bound: the constant is + // env-overridable and production applies no clamp, so pinning an arbitrary + // ceiling here would fail a validly configured node + // (`DKG_CATCHUP_MAX_CONCURRENT_PEERS=32`) while proving nothing about the + // code. The real contract is the parse: a positive integer, else the + // default. expect(Number.isInteger(CATCHUP_MAX_CONCURRENT_PEER_SYNCS)).toBe(true); + expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); }); }); diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index d2f684ea17..e863fcf991 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -3,6 +3,8 @@ import { CATCHUP_BACKPRESSURE_BASE_DELAY_MS, CATCHUP_BACKPRESSURE_MAX_DELAY_MS, CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + resolveCatchupBackpressureMaxWaitMs, FOREGROUND_CATCHUP_SYNC_PRIORITY, nextCatchupBackpressureDelayMs, runCatchupPlaneWithPolicy, @@ -267,3 +269,68 @@ describe('CATCHUP_BACKPRESSURE_MAX_WAIT_MS', () => { expect(clock.now()).toBeLessThanOrEqual(CATCHUP_BACKPRESSURE_MAX_WAIT_MS); }); }); + +describe('resolveCatchupBackpressureMaxWaitMs', () => { + it('treats a blank assignment as unset, not as zero', () => { + // `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS=` in a compose file / .env / unit + // file is the normal shape for "not set". `Number('')` is 0, so a naive + // parser turns it into "never retry" — silently worse than the fixed ladder + // this policy replaced, and with no log to notice it by. + expect(resolveCatchupBackpressureMaxWaitMs('')).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + expect(resolveCatchupBackpressureMaxWaitMs(' ')).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + expect(resolveCatchupBackpressureMaxWaitMs(undefined)).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); + + it('honours an explicit zero as "do not retry"', () => { + expect(resolveCatchupBackpressureMaxWaitMs('0')).toBe(0); + expect(resolveCatchupBackpressureMaxWaitMs(' 0 ')).toBe(0); + }); + + it('honours a positive integer budget', () => { + expect(resolveCatchupBackpressureMaxWaitMs('45000')).toBe(45_000); + expect(resolveCatchupBackpressureMaxWaitMs(' 600000 ')).toBe(600_000); + }); + + it('falls back to the default for values it cannot honour', () => { + for (const raw of ['-1', '1.5', 'abc', 'NaN', 'Infinity', '1e3ms']) { + expect(resolveCatchupBackpressureMaxWaitMs(raw)).toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + } + }); + + it('resolves the module constant through the same parser', () => { + expect(CATCHUP_BACKPRESSURE_MAX_WAIT_MS).toBe( + resolveCatchupBackpressureMaxWaitMs(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS), + ); + }); +}); + +describe('foreground backoff timer', () => { + it('unrefs the pending sleep so a backoff cannot outlive agent.stop()', async () => { + // The budget is minutes now. A referenced timer would hold the event loop + // open for that long past shutdown, and every other test injects `wait`, + // so the real `defaultWait` would otherwise never be exercised. + const realSetTimeout = globalThis.setTimeout; + const unref = vi.fn(); + const spy = vi.spyOn(globalThis, 'setTimeout').mockImplementation((( + handler: TimerHandler, + timeout?: number, + ...rest: unknown[] + ) => { + const handle = (realSetTimeout as any)(handler, Math.min(timeout ?? 0, 1), ...rest); + return Object.assign(handle as object, { unref }) as never; + }) as never); + + try { + let attempts = 0; + await runCatchupPlaneWithPolicy('foreground', async () => { + attempts += 1; + return { deferredBackpressure: attempts === 1 ? 1 : 0 }; + }, { retry: { maxWaitMs: 5_000 } }); + + expect(attempts).toBe(2); + expect(unref).toHaveBeenCalled(); + } finally { + spy.mockRestore(); + } + }); +}); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 8c31cea4ed..d951d87595 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -88,6 +88,12 @@ function emptyShared() { async function runCatchup(request: CatchupRunRequest): Promise { const prepared = await invoke<{ preferredPeerId?: string; + /** + * The preferred peer ONLY when it came from authoritative Context Graph + * metadata. A join-approval bootstrap hint arrives as `preferredPeerId` + * without this, so it orders the walk but can never end it. + */ + authoritativePeerId?: string; isPrivateContextGraph: boolean; peerIds: string[]; connectedPeers: number; @@ -209,7 +215,8 @@ async function runCatchup(request: CatchupRunRequest): Promise const needDurable = !optimize || !authorityProven.durable; const needSharedMemory = request.includeSharedMemory && (!optimize || !authorityProven.sharedMemory); - const fromAuthority = peerId === prepared.preferredPeerId; + const fromAuthority = prepared.authoritativePeerId !== undefined + && peerId === prepared.authoritativePeerId; if (!needDurable) { const shared = needSharedMemory ? await runCatchupPlaneWithPolicy('foreground', syncSharedMemory) @@ -358,8 +365,8 @@ async function runCatchup(request: CatchupRunRequest): Promise // add a round-trip to the front of every round, with no early stop to earn it // back. In that case the walk opens at the full concurrency cap — the previous // first-round latency. - const authorityFirst = prepared.preferredPeerId !== undefined - && syncCapable[0] === prepared.preferredPeerId; + const authorityFirst = prepared.authoritativePeerId !== undefined + && syncCapable[0] === prepared.authoritativePeerId; const waveSizes = CATCHUP_STOP_ON_PROOF ? catchupWaveSizes( syncCapable.length, diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 48845862f2..e478be9422 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -822,6 +822,13 @@ class WorkerCatchupRunner implements CatchupRunner { const [contextGraphId] = args as [string]; const isPrivateContextGraph = await agent.isPrivateContextGraph(contextGraphId); const preferredPeerId = await agent.resolvePreferredSyncPeerId(contextGraphId); + // Ranking uses the preferred peer; letting ONE peer's answer stand for + // the whole graph requires the stricter notion. A join-approval + // bootstrap hint is authenticated but can be stale, so it orders the + // walk without being allowed to end it. + const authoritativePeerId = typeof agent.resolveAuthoritativeSyncPeerId === 'function' + ? await agent.resolveAuthoritativeSyncPeerId(contextGraphId) + : undefined; if (preferredPeerId) { await agent.ensurePeerConnected(preferredPeerId); } @@ -837,6 +844,7 @@ class WorkerCatchupRunner implements CatchupRunner { return { preferredPeerId, + authoritativePeerId, isPrivateContextGraph, peerIds, connectedPeers: peerIds.length, diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 47cfd9803c..67da244e19 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -146,7 +146,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const result = await runWorkerCatchup({ contextGraphId: 'cg-one-payload', includeSharedMemory: true }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': probeOrder.push(args[0] as string); return true; @@ -335,7 +335,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const result = await runWorkerCatchup({ contextGraphId: 'cg-durable-only', includeSharedMemory: false }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': return true; case 'syncDurable': @@ -364,7 +364,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-swm', includeSharedMemory: true }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': return true; case 'syncDurable': @@ -408,7 +408,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-fallback', includeSharedMemory: true }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': return true; case 'syncDurable': @@ -464,6 +464,41 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.dataSynced).toBe(1); }); + it('does not let a bootstrap-hint preferred peer stop the walk', async () => { + // `resolvePreferredSyncPeerId` falls back to the authenticated join-approval + // hint when metadata resolves no curator. That hint can be stale — a curator + // that has since rotated its libp2p identity leaves an ordinary member on + // that peer id — so it orders the walk but must never end it. The worker + // sees that as `preferredPeerId` WITHOUT `authoritativePeerId`. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-hint-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + }); + it('opens at the full concurrency cap when no curator resolved', async () => { // A single-peer opening wave buys "one payload from the curator". Without a // resolvable curator it buys nothing, so the walk must not serialise the @@ -540,7 +575,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-only', includeSharedMemory: true }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': return true; case 'syncDurable': @@ -623,7 +658,17 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) // unrelated peer that has never heard of the graph answers empty. On the // wire those two peers are indistinguishable, so the empty answer must not // stop the walk and must not settle the job as `done`. - const peerIds = ['peer-empty', 'peer-data-failed', 'peer-quiet']; + // + // The empty peers fill the ENTIRE first wave and the data-bearing peer sits + // behind a wave boundary, so a regression that accepted any clean-empty + // round as proof would stop before ever reaching it. A same-wave setup + // could not observe that. + const emptyPeers = Array.from( + { length: CATCHUP_MAX_CONCURRENT_PEER_SYNCS }, + (_, i) => `peer-empty-${i}`, + ); + const peerIds = [...emptyPeers, 'peer-data-failed', 'peer-quiet']; + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); const durableCalls: string[] = []; const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-mask', includeSharedMemory: false }, async (method, args) => { @@ -663,11 +708,15 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) } }); - // Emptiness is never a stop condition, so every peer is still contacted. + // Emptiness is never a stop condition, so the walk crosses the wave + // boundary and still reaches the data-bearing peer. expect(durableCalls).toEqual(peerIds); + expect(durableCalls).toContain('peer-data-failed'); + expect(result.peersNotAttempted).toBe(0); expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBe(0); // The clean-empty peers are still recorded as clean empty completions… - expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(2); + expect(result.cleanPlaneCompletions?.durable.emptyPeers) + .toBe(peerIds.length - 1); // …but the round fetched data and failed, so readiness must not follow. expect(result.diagnostics?.durable.fetchedDataTriples).toBe(5_000); expect(result.diagnostics?.durable.failedPhases).toBe(1); diff --git a/packages/cli/test/catchup-runner-worker-killswitch.test.ts b/packages/cli/test/catchup-runner-worker-killswitch.test.ts index 07ff65395a..883bc05651 100644 --- a/packages/cli/test/catchup-runner-worker-killswitch.test.ts +++ b/packages/cli/test/catchup-runner-worker-killswitch.test.ts @@ -125,7 +125,7 @@ describe('catch-up progressive walk kill-switch', () => { const result = await runWorkerCatchup({ contextGraphId: 'cg-killswitch', includeSharedMemory: true }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; case 'waitForSyncProtocol': return true; case 'syncDurable': { diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index f6a26cd343..016a47592e 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -82,6 +82,24 @@ describe('WorkerCatchupRunner lifecycle', () => { expect((outcome as Error).message).toContain('exited'); }); + it('fails a run started after the worker died instead of posting into the void', async () => { + // The pending-run case alone is not enough: the runner is constructed once + // per daemon, and `postMessage` to a dead worker neither throws nor + // delivers, so without the latch every LATER subscribe hung too. + const runner = createCatchupRunner(stubAgent); + await runner.close(); + const postedBeforeLaterRun = workerControl.state.posted.length; + + const later = runner.run({ contextGraphId: 'cg-later', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + + const outcome = await withinTick(later); + expect(outcome).toBeInstanceOf(Error); + expect((outcome as Error).message).toContain('exited'); + // …and it must not have queued work onto the dead worker. + expect(workerControl.state.posted).toHaveLength(postedBeforeLaterRun); + }); + it('rejects every pending run exactly once', async () => { const runner = createCatchupRunner(stubAgent); const first = runner.run({ contextGraphId: 'cg-a', includeSharedMemory: false }) From c141370f2795f1115c09db24616cb8f9641fbb53 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 10:09:14 +0200 Subject: [PATCH 13/44] fix(sync): make the removed retryDelaysMs option a compile error, not a silent no-op MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third raise of the removed-retry-API thread landed a sub-point the earlier two did not: removing the CONSTANT is loud (a compile error), but the `retryDelaysMs` OPTION would have been silently ignored when passed via a variable rather than an object literal — turning an intended `retryDelaysMs: [10]` into a wait of up to the full 180 s budget, with no signal at all. Retained on the options type as `retryDelaysMs?: never` with a doc comment naming the replacements, so any caller still setting it fails to compile. Verified with a throwaway probe: passing it through a variable now errors (TS2345) where it previously compiled and was dropped. Still no alias for the constant — an alias could only export a schedule the node no longer follows — and the CHANGELOG now records both halves. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- packages/agent/src/sync/catchup-policy.ts | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ac47341f7c..23e67dbb47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Removed -- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. +- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. `retryDelaysMs` is retained on the options type as `never`, so a caller that still sets it fails to compile rather than having it silently ignored — an ignored `retryDelaysMs: [10]` would otherwise turn an intended 10 ms schedule into a wait of up to the full budget. ### Operator knobs diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 342711432b..0fd24c7d54 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -72,6 +72,21 @@ export interface CatchupBackpressureRetryPolicy { /** Deterministic seams for tests; never operator configuration. */ export interface CatchupPlanePolicyClock { + /** + * Removed with the fixed `[100, 250, 500]` ladder it configured. + * + * Declared as `never` rather than deleted outright so a caller that still + * sets it FAILS TO COMPILE instead of having it silently ignored — an + * ignored `retryDelaysMs: [10]` would turn an intended 10 ms schedule into a + * wait of up to `CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, which is a much worse way + * to learn about the change than a type error. + * + * Operators: use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`. Tests: use `retry` + * with `now` / `wait` / `random`. + * + * @deprecated + */ + retryDelaysMs?: never; retry?: CatchupBackpressureRetryPolicy; wait?: (delayMs: number) => Promise; now?: () => number; From 4f03a5f1ffec93e9467d3a9716901074781c07d3 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 10:55:00 +0200 Subject: [PATCH 14/44] fix(sync): an empty curator round must not settle a private plane (review round 6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 🔴 Readiness deliberately refuses to prove a PRIVATE plane from an empty response, but the walk let the curator's clean-empty round settle one anyway. A private graph whose curator answered empty therefore stopped before any fallback peer that may hold authorized private data, turning a recoverable catch-up into `unreachable`. Emptiness now settles public planes only; a verified private-only response is content, not emptiness, and still counts. Both directions tested, mutation-checked. * 🔴 Every worker test stubbed `prepareCatchup`, so the parent-side bridge that actually derives `authoritativePeerId` and forwards `source` was never exercised — a regression returning the bootstrap hint as the authority, or dropping `source` before `syncFromPeerDetailed`, would have left them green. Added agent-bridge tests that drive real `invoke` messages through `WorkerCatchupRunner` with a stub agent. Mutation-checked: both regressions now fail. * 🟡 Authority provenance was inferred from `resolveCuratorPeerId`'s cache-eviction side effect. It is now an explicit result — `classifySyncPeerProvenance(hint, curator)`, pure, with the hint captured BEFORE resolution — so the distinction that decides whether one peer may stand for a whole graph is a return value rather than a non-local invariant two methods have to agree on. `cg-resolve-refresh` caught the first attempt at this breaking a documented contract; the classifier is now independent of any sibling method, which keeps that test meaningful. * 🟡 Trimmed the agent barrel to what a cross-package consumer needs. The backoff curve, the env parser and the injected clock seams are retry-policy internals; in-package tests import them from the source path, so they no longer sit on the published surface. Co-Authored-By: Claude Opus 5 (1M context) --- packages/agent/src/dkg-agent-lifecycle.ts | 80 ++++++++++++----- packages/agent/src/index.ts | 12 +-- packages/agent/test/sync-policy.test.ts | 41 +++++++++ .../cli/src/catchup-runner-worker-impl.ts | 27 ++++-- .../test/catchup-runner-worker-impl.test.ts | 88 +++++++++++++++++++ .../catchup-runner-worker-lifecycle.test.ts | 87 ++++++++++++++++++ 6 files changed, 299 insertions(+), 36 deletions(-) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 729ff443ed..dee073a4e8 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -1160,6 +1160,28 @@ function emptySwmRecoveryResult(): RecoverContextGraphSwmResult { }; } +/** Where a resolved catch-up sync peer came from; see `resolveSyncPeerWithProvenance`. */ +export interface SyncPeerResolution { + peerId?: string; + provenance: 'metadata' | 'bootstrap-hint' | 'none'; +} + +/** + * Classify a resolved curator against the join-approval hint captured BEFORE + * resolution. Pure, so the distinction that decides whether one peer may stand + * for a whole Context Graph does not depend on any cache side effect. + */ +export function classifySyncPeerProvenance( + bootstrapHint: string | undefined, + curatorPeerId: string | undefined, +): SyncPeerResolution { + if (curatorPeerId && curatorPeerId !== bootstrapHint) { + return { peerId: curatorPeerId, provenance: 'metadata' }; + } + const peerId = curatorPeerId ?? bootstrapHint; + return peerId ? { peerId, provenance: 'bootstrap-hint' } : { provenance: 'none' }; +} + export class LifecycleSyncMethods extends DKGAgentBase { async runContextGraphSyncWithBackpressure(this: DKGAgent, ctx: OperationContext, @@ -6298,40 +6320,54 @@ export class LifecycleSyncMethods extends DKGAgentBase { return orderCatchupPeers(peers, preferredPeerId, privateOnly, this.knownCorePeerIds); } + /** + * Resolve the catch-up sync peer together with WHERE it came from. + * + * The distinction is load-bearing, so it is a return value rather than + * something a caller has to infer: only a metadata-resolved curator may let + * one peer's answer stand for the whole graph. The authenticated + * join-approval hint is a fine ranking signal but can be stale — peer ids are + * cryptographic identities, so a curator that has rotated its libp2p key + * leaves an ordinary member sitting on the id the hint still names. + * + * The bootstrap hint is captured BEFORE resolution so this does not depend on + * `resolveCuratorPeerId`'s cache-eviction side effect; the only property + * relied on is its documented contract, that it either returns a + * metadata-derived curator or echoes that same hint back. + */ + async resolveSyncPeerWithProvenance(this: DKGAgent, contextGraphId: string): Promise { + const bootstrapHint = this.preferredSyncPeers.get(contextGraphId); + return classifySyncPeerProvenance( + bootstrapHint, + await this.resolveCuratorPeerId(contextGraphId), + ); + } + async resolvePreferredSyncPeerId(this: DKGAgent, contextGraphId: string): Promise { - // resolveCuratorPeerId consults authoritative metadata first and only then - // falls back to the authenticated join-approval hint. Calling it before - // reading preferredSyncPeers prevents that bootstrap hint from pinning all - // later catchups to a curator that metadata has superseded. - const curatorPeerId = await this.resolveCuratorPeerId(contextGraphId); - return curatorPeerId ?? this.preferredSyncPeers.get(contextGraphId); + // Ranking takes the best peer available, whatever its provenance. Kept + // independent of the sibling method so this stays exercisable on its own. + const bootstrapHint = this.preferredSyncPeers.get(contextGraphId); + return classifySyncPeerProvenance( + bootstrapHint, + await this.resolveCuratorPeerId(contextGraphId), + ).peerId; } /** - * The preferred sync peer, but only when it came from authoritative Context - * Graph metadata rather than the bootstrap join-approval hint. - * - * `resolveCuratorPeerId` deletes the hint the moment metadata resolves a - * curator, so a returned peer that is STILL the live hint means metadata did - * not resolve one and we are looking at a bootstrap value that can be stale - * (a curator that has since rotated its libp2p identity, leaving an ordinary - * member on that peer id). + * The sync peer ONLY when it is a metadata-resolved curator. * - * Callers that merely want to try the best peer first should keep using + * Callers that merely want to try the best peer first should use * {@link resolvePreferredSyncPeerId}. Only callers that let one peer's answer * stand for the whole graph — the foreground catch-up walk's early stop — - * need this stricter notion, because a non-curator that happens to be ranked - * first must never be able to cut the walk short. + * need this stricter notion, because a peer that happens to be ranked first + * must never be able to cut the walk short. */ async resolveAuthoritativeSyncPeerId( this: DKGAgent, contextGraphId: string, ): Promise { - const preferredPeerId = await this.resolvePreferredSyncPeerId(contextGraphId); - if (!preferredPeerId) return undefined; - return preferredPeerId === this.preferredSyncPeers.get(contextGraphId) - ? undefined - : preferredPeerId; + const resolved = await this.resolveSyncPeerWithProvenance(contextGraphId); + return resolved.provenance === 'metadata' ? resolved.peerId : undefined; } async ensurePeerConnected(this: DKGAgent, peerId: string): Promise { diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index 496905ab49..a9c2fb25cb 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -314,17 +314,17 @@ export { CATCHUP_STOP_ON_PROOF, catchupWaveSizes, } from './sync/catchup-concurrency.js'; +// Only what a cross-package consumer genuinely needs. The CLI daemon's Worker +// catch-up runner drives the same plane policy and must not deep-import the +// compiled `dist/`; everything else here — the backoff curve, the env parser, +// the injected clock seams — is retry-policy internals, and in-package tests +// import those from `./sync/catchup-policy.js` directly rather than pinning +// them to the published surface. export { - CATCHUP_BACKPRESSURE_BASE_DELAY_MS, - CATCHUP_BACKPRESSURE_JITTER_RATIO, - CATCHUP_BACKPRESSURE_MAX_DELAY_MS, CATCHUP_BACKPRESSURE_MAX_WAIT_MS, - DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, - resolveCatchupBackpressureMaxWaitMs, FOREGROUND_CATCHUP_SYNC_PRIORITY, catchupPriorityForMode, catchupSourceForMode, - nextCatchupBackpressureDelayMs, runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, type CatchupAdmissionSource, diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 04e868e40f..39bf98d0e7 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -9,6 +9,7 @@ import { syncPriorityClass, validateSyncResponderSnapshotLimitsConfig, } from '../src/sync/policy.js'; +import { classifySyncPeerProvenance } from '../src/dkg-agent-lifecycle.js'; describe('sync Context Graph policy', () => { it('normalizes safe integer priorities and preserves stable input order for ties', () => { @@ -88,3 +89,43 @@ describe('normalizeSyncAdmissionSource', () => { } }); }); + +describe('classifySyncPeerProvenance', () => { + const HINT = '12D3KooWBootstrapHint'; + const CURATOR = '12D3KooWMetadataCurator'; + + it('marks a metadata-resolved curator as authoritative', () => { + expect(classifySyncPeerProvenance(undefined, CURATOR)) + .toEqual({ peerId: CURATOR, provenance: 'metadata' }); + expect(classifySyncPeerProvenance(HINT, CURATOR)) + .toEqual({ peerId: CURATOR, provenance: 'metadata' }); + }); + + it('marks an echoed bootstrap hint as NOT authoritative', () => { + // `resolveCuratorPeerId` echoes the join-approval hint when metadata + // resolves no curator. That hint can be stale — peer ids are cryptographic + // identities, so a curator that has rotated its libp2p key leaves an + // ordinary member on the id the hint still names — so it may rank the walk + // but must never let one peer stand for the whole graph. + expect(classifySyncPeerProvenance(HINT, HINT)) + .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + expect(classifySyncPeerProvenance(HINT, undefined)) + .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + }); + + it('reports no peer when neither source produced one', () => { + expect(classifySyncPeerProvenance(undefined, undefined)) + .toEqual({ provenance: 'none' }); + }); + + it('keeps ranking availability identical to authority eligibility only for metadata', () => { + // The ranking caller takes `.peerId` regardless of provenance; the + // early-stop caller takes it only for 'metadata'. Pin that they differ + // exactly on the hint case. + for (const [hint, curator] of [[HINT, HINT], [HINT, undefined]] as const) { + const resolved = classifySyncPeerProvenance(hint, curator); + expect(resolved.peerId).toBe(HINT); + expect(resolved.provenance).not.toBe('metadata'); + } + }); +}); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index d951d87595..abbab50b56 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -190,6 +190,23 @@ async function runCatchup(request: CatchupRunRequest): Promise const authorityProvedEverything = (): boolean => authorityProven.durable && (!request.includeSharedMemory || authorityProven.sharedMemory); + /** + * Whether the curator's round settles a plane well enough to stop walking. + * + * Verified content always does. A clean EMPTY round only does for a PUBLIC + * graph: readiness deliberately refuses to prove a private plane from an + * empty response, so stopping on one would strand the walk without proving + * anything — skipping fallback peers that may hold authorized private data + * and turning a recoverable catch-up into `unreachable`. A verified + * private-only response is content, not emptiness, and still counts. + */ + const authoritySettles = (evidence: { + verifiedDataPeers: number; + verifiedPrivateOnlyPeers?: number; + emptyPeers: number; + }): boolean => catchupPlaneProvenByData(evidence) + || (!prepared.isPrivateContextGraph && evidence.emptyPeers > 0); + // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we // can from the other peers instead of failing the entire subscribe/catch-up. const syncPeer = async (peerId: string): Promise => { @@ -269,10 +286,7 @@ async function runCatchup(request: CatchupRunRequest): Promise // The positive half runs through the SAME predicate the readiness // classifier uses, just applied to one peer's evidence rather than the // round's, so the stop condition cannot drift from the readiness rule. - if ( - fromAuthority - && (catchupPlaneProvenByData(durableEvidence) || durableEvidence.emptyPeers > 0) - ) { + if (fromAuthority && authoritySettles(durableEvidence)) { authorityProven.durable = true; } } @@ -307,10 +321,7 @@ async function runCatchup(request: CatchupRunRequest): Promise // graph that has durable data, and `includeSharedMemory` defaults to // true on subscribe, so without this the early stop would almost never // fire in the shape the fix targets. - if ( - fromAuthority - && (catchupPlaneProvenByData(sharedEvidence) || sharedEvidence.emptyPeers > 0) - ) { + if (fromAuthority && authoritySettles(sharedEvidence)) { authorityProven.sharedMemory = true; } } diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 67da244e19..dc7b41d3d2 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -464,6 +464,94 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.dataSynced).toBe(1); }); + it('does not let an empty curator round settle a PRIVATE plane', async () => { + // Readiness deliberately refuses to prove a private plane from an empty + // response, so stopping the walk on one would strand it: fallback peers + // that may hold authorized private data are skipped and a recoverable + // catch-up turns into `unreachable`. Emptiness only settles public planes. + const peerIds = ['peer-curator', 'peer-with-data', 'peer-c', 'peer-d', 'peer-e', 'peer-f']; + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-empty', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + if (args[0] === 'peer-curator') { + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The authorized fallback peer must still be reached. + expect(durableCalls).toContain('peer-with-data'); + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBeGreaterThan(0); + }); + + it('still lets a verified private-only curator round settle a private plane', async () => { + // The complement: a cryptographically verified V2 response whose public + // graph is intentionally empty is CONTENT, not emptiness, and must keep + // working as positive proof on a private graph. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 8, + fetchedMetaTriples: 8, + fetchedDataTriples: 0, + insertedMetaTriples: 8, + insertedDataTriples: 0, + verifiedPrivateOnlyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.durable.verifiedPrivateOnlyPeers).toBe(1); + }); + it('does not let a bootstrap-hint preferred peer stop the walk', async () => { // `resolvePreferredSyncPeerId` falls back to the authenticated join-approval // hint when metadata resolves no curator. That hint can be stale — a curator diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index 016a47592e..020e0374f1 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -33,6 +33,11 @@ const workerControl = vi.hoisted(() => { state.listeners.set(event, existing); } + /** Test-only: deliver a message as if the worker had sent it. */ + static emitToRunner(message: unknown) { + for (const listener of state.listeners.get('message') ?? []) listener(message); + } + postMessage(message: unknown) { state.posted.push(message); } @@ -115,3 +120,85 @@ describe('WorkerCatchupRunner lifecycle', () => { .resolves.toEqual(['rejected', 'rejected']); }); }); + +/** + * The parent-side bridge (`WorkerCatchupRunner.invokeAgent`) is the boundary + * every worker-impl test stubs out — those tests supply `authoritativePeerId` + * and observe `source` themselves, so a regression HERE would leave them green + * while production early-stopped on a stale bootstrap hint or reported + * foreground catch-up as `unspecified` in scheduler diagnostics. + */ +describe('WorkerCatchupRunner agent bridge', () => { + function bridgeAgent(overrides: Record = {}) { + const calls: Record = { durable: [], shared: [] }; + const agent = { + isPrivateContextGraph: async () => false, + resolvePreferredSyncPeerId: async () => 'peer-hint', + resolveAuthoritativeSyncPeerId: async () => undefined, + ensurePeerConnected: async () => {}, + primeCatchupConnections: async () => {}, + selectCatchupPeers: (peers: Array<{ toString(): string }>) => peers, + node: { libp2p: { getConnections: () => [] } }, + syncFromPeerDetailed: async (...args: unknown[]) => { + calls.durable.push(args); + return {}; + }, + syncSharedMemoryFromPeerDetailed: async (...args: unknown[]) => { + calls.shared.push(args); + return {}; + }, + ...overrides, + }; + return { agent: agent as unknown as DKGAgent, calls }; + } + + /** Drive one `invoke` through the real bridge and return what it posted back. */ + async function invokeThroughBridge(agent: DKGAgent, method: string, args: unknown[]) { + createCatchupRunner(agent); + const before = workerControl.state.posted.length; + workerControl.FakeWorker.emitToRunner({ type: 'invoke', invokeId: 1, method, args }); + for (let i = 0; i < 50 && workerControl.state.posted.length === before; i += 1) { + await new Promise((resolve) => setTimeout(resolve, 1)); + } + return workerControl.state.posted.at(-1) as { result?: any; error?: string }; + } + + it('does not report a bootstrap-hint peer as the catch-up authority', async () => { + const { agent } = bridgeAgent(); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-hint']); + + // The hint still ranks the walk… + expect(posted.result.preferredPeerId).toBe('peer-hint'); + // …but must not be handed to the worker as an authority. + expect(posted.result.authoritativePeerId).toBeUndefined(); + }); + + it('reports a metadata-resolved curator as the catch-up authority', async () => { + const { agent } = bridgeAgent({ + resolvePreferredSyncPeerId: async () => 'peer-curator', + resolveAuthoritativeSyncPeerId: async () => 'peer-curator', + }); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-meta']); + + expect(posted.result.preferredPeerId).toBe('peer-curator'); + expect(posted.result.authoritativePeerId).toBe('peer-curator'); + }); + + it('forwards the admission source into both detailed sync calls', async () => { + const { agent, calls } = bridgeAgent(); + + await invokeThroughBridge(agent, 'syncDurable', ['peer-a', 'cg-x', 2000, 'catchup-foreground']); + await invokeThroughBridge(agent, 'syncSharedMemory', ['peer-a', 'cg-x', 2000, 'catchup-foreground']); + + expect(calls.durable).toHaveLength(1); + expect(calls.durable[0]!.at(-1)).toMatchObject({ + priority: 2000, + source: 'catchup-foreground', + }); + expect(calls.shared).toHaveLength(1); + expect(calls.shared[0]!.at(-1)).toMatchObject({ + priority: 2000, + source: 'catchup-foreground', + }); + }); +}); From 6683d9c7fa761c9f0e56225f049be63e94c6ad4b Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 10:59:21 +0200 Subject: [PATCH 15/44] fix(sync): actually type the catch-up plane boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I told review round 2 this landed in def08c3a5. It did not — `git show` on every commit since shows zero occurrences of `CatchupDurableResult`. I verified the edit from my working tree and a passing typecheck without confirming it reached the commit, which is exactly the mistake the reviewer's re-raise caught. `PeerRound` now carries `CatchupDurableResult | null` / `CatchupSharedMemoryResult | null` derived from the agent's `DurableSyncResult` / `SharedMemorySyncResult`, the two RPC helpers take `CatchupPlaneContext` and return typed results instead of `invoke`, and `emptyShared()` is annotated so a missing counter is a compile error rather than an `undefined` reaching the accumulator. The only remaining `any` is the pre-existing `parentPort` message plumbing. Verified this time by grep against the committed tree, not the worktree. Co-Authored-By: Claude Opus 5 (1M context) --- .../cli/src/catchup-runner-worker-impl.ts | 40 ++++++++++++++----- 1 file changed, 31 insertions(+), 9 deletions(-) diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index abbab50b56..535cb1cf30 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -7,6 +7,9 @@ import { mapWithConcurrency, runCatchupPlaneWithPolicy, runCatchupPlanesWithPolicy, + type CatchupPlaneContext, + type DurableSyncResult, + type SharedMemorySyncResult, } from '@origintrail-official/dkg-agent'; import { addCatchupPlaneEvidence, @@ -55,16 +58,31 @@ parentPort!.on('message', async (message: any) => { } }); -/** A per-peer sync round; a plane is absent when the walk skipped it. */ +/** + * One peer's durable plane. The wire shape is the agent's own + * `DurableSyncResult`, structured-cloned back across the Worker RPC; + * `verifiedPrivateOnlyResponses` is normalized to a number on arrival so the + * accumulation below never has to re-guard it. + */ +type CatchupDurableResult = DurableSyncResult & { verifiedPrivateOnlyResponses: number }; + +/** One peer's shared-memory plane, as returned across the Worker RPC. */ +type CatchupSharedMemoryResult = SharedMemorySyncResult; + +/** + * One peer's sync round. A plane is `null` when the walk deliberately skipped + * it because the authority already settled that plane — that is the ONLY + * exceptional case, and it is distinct from a plane that ran and failed. + */ interface PeerRound { peerId: string; /** The resolved curator for this Context Graph produced this round. */ fromAuthority: boolean; - durable: any | null; - shared: any | null; + durable: CatchupDurableResult | null; + shared: CatchupSharedMemoryResult | null; } -function emptyShared() { +function emptyShared(): CatchupSharedMemoryResult { return { insertedTriples: 0, fetchedMetaTriples: 0, @@ -210,15 +228,19 @@ async function runCatchup(request: CatchupRunRequest): Promise // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we // can from the other peers instead of failing the entire subscribe/catch-up. const syncPeer = async (peerId: string): Promise => { - const syncDurable = ({ priority, source }: { priority?: number; source?: string }) => - invoke('syncDurable', peerId, request.contextGraphId, priority, source) + const syncDurable = ( + { priority, source }: CatchupPlaneContext, + ): Promise => + invoke('syncDurable', peerId, request.contextGraphId, priority, source) .catch(() => createFailedPeerDurableSyncResult()) - .then((rawDurable: any) => ({ + .then((rawDurable) => ({ ...rawDurable, verifiedPrivateOnlyResponses: rawDurable.verifiedPrivateOnlyResponses ?? 0, })); - const syncSharedMemory = ({ priority, source }: { priority?: number; source?: string }) => - invoke('syncSharedMemory', peerId, request.contextGraphId, priority, source) + const syncSharedMemory = ( + { priority, source }: CatchupPlaneContext, + ): Promise => + invoke('syncSharedMemory', peerId, request.contextGraphId, priority, source) .catch(() => emptyShared()); // Narrow each fallback peer to the planes the AUTHORITY has not already From 8cd3cef79df116f747410a1ab2acbd265ddbf2fb Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 11:46:10 +0200 Subject: [PATCH 16/44] test(sync): stop leaking catch-up env overrides into sibling suites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both new test files set `process.env` from `vi.hoisted` so the module-load-time constants pick them up — but that mutates the REAL process env, and vitest can reuse a worker process across files within a CI shard. `cli [3/4]` contains both the kill-switch suite and `daemon-http-behavior-extra`, which SPAWNS a real daemon that inherits the parent environment, so the override could reach a process that never asked for it. Each file now captures the previous value in the same hoisted block and restores it in `afterAll`. Note on the shard failure this came from: `daemon-http-behavior-extra`'s SIGTERM case failed with "Daemon did not become ready within 45s" on cli [3/4]. It passes locally at HEAD in 12.8s against a real daemon, so boot is not broken; this removes the one cross-file interaction my change actually introduced there. Co-Authored-By: Claude Opus 5 (1M context) --- .../cli/test/catchup-runner-worker-impl.test.ts | 15 +++++++++++++-- .../test/catchup-runner-worker-killswitch.test.ts | 15 +++++++++++++-- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index dc7b41d3d2..9bee90395b 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -14,7 +14,7 @@ // transfers one payload instead of one per peer. A non-authoritative peer's // clean round settles nothing — it can neither stop the walk nor narrow a // later peer to one plane. -import { describe, expect, it, vi } from 'vitest'; +import { afterAll, describe, expect, it, vi } from 'vitest'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, FOREGROUND_CATCHUP_SYNC_PRIORITY, @@ -25,8 +25,19 @@ import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner. // for this file so the persistently-deferred case settles quickly; the exact // deadline arithmetic is pinned deterministically in // packages/agent/test/catchup-policy.test.ts with an injected clock. -vi.hoisted(() => { +// `vi.hoisted` runs before imports so the module-load-time constant picks this +// up — but it mutates the REAL process env, and vitest can reuse a worker +// process across files in a shard. Anything loaded afterwards, including a +// daemon spawned by a sibling suite, would otherwise inherit the shortened backpressure budget. +const previousCATCHUPBACKPRESSUREMAXWAITMS = vi.hoisted(() => { + const before = process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = '250'; + return before; +}); + +afterAll(() => { + if (previousCATCHUPBACKPRESSUREMAXWAITMS === undefined) delete process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; + else process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = previousCATCHUPBACKPRESSUREMAXWAITMS; }); // The worker impl wires itself to `parentPort` at module load, so a diff --git a/packages/cli/test/catchup-runner-worker-killswitch.test.ts b/packages/cli/test/catchup-runner-worker-killswitch.test.ts index 883bc05651..a25e76102f 100644 --- a/packages/cli/test/catchup-runner-worker-killswitch.test.ts +++ b/packages/cli/test/catchup-runner-worker-killswitch.test.ts @@ -5,12 +5,23 @@ // them, and no early stop — so an operator can back the optimisation out // without a redeploy if a graph ever lands short. The switch is read once at // module load, which is why this lives in its own file. -import { describe, expect, it, vi } from 'vitest'; +import { afterAll, describe, expect, it, vi } from 'vitest'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS } from '@origintrail-official/dkg-agent'; import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner.js'; -vi.hoisted(() => { +// `vi.hoisted` runs before imports so the module-load-time constant picks this +// up — but it mutates the REAL process env, and vitest can reuse a worker +// process across files in a shard. Anything loaded afterwards, including a +// daemon spawned by a sibling suite, would otherwise inherit the kill-switch. +const previousCATCHUPSTOPONPROOF = vi.hoisted(() => { + const before = process.env.DKG_CATCHUP_STOP_ON_PROOF; process.env.DKG_CATCHUP_STOP_ON_PROOF = '0'; + return before; +}); + +afterAll(() => { + if (previousCATCHUPSTOPONPROOF === undefined) delete process.env.DKG_CATCHUP_STOP_ON_PROOF; + else process.env.DKG_CATCHUP_STOP_ON_PROOF = previousCATCHUPSTOPONPROOF; }); const fakeParentPort = vi.hoisted(() => { From 7021150e9fd1d7f2a84ee712f529558f3fad844d Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 12:31:58 +0200 Subject: [PATCH 17/44] fix(sync): curator provenance from the resolver, and let it prove an empty graph MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 7. Provenance was derived by comparing the resolved curator against the join-approval hint captured before resolution. That reads the ORDINARY case wrong: the join approval normally comes from the curator, so both routes name the same peer, and "metadata confirmed the curator" became indistinguishable from "metadata found nothing and the hint was echoed back". The catch-up walk therefore refused to treat a real curator as authority exactly where the early stop is worth the most. `resolveCuratorSyncPeer` now returns the provenance it actually took, and the classifier that could not know it is gone. A registered public graph with no Knowledge Assets still serves its own `/_meta` definition triples, so its host answers metadata-only, never wire-empty — no whole-round emptiness rule could ever fire for it and subscribe reported `unreachable` forever. The CURATOR's hosted-empty round now proves it. Scoped to the curator deliberately: accepting any peer's metadata-only round would resettle #2006 itself, since a member holding `_meta` but no data yet is the commonest state on the network. The retry deadline was taken after the first admission attempt, making the budget "however long that attempt took, PLUS maxWaitMs". It is now taken before it. The budget bounds how long a plane keeps ASKING; it does not preempt a round the scheduler already accepted, and the docs now say so rather than implying an absolute cap the policy cannot enforce. Tests drive the REAL resolver rather than a stub of it, cover the private shared-memory plane's empty rule, and pin the removed `retryDelaysMs` contract in an enforced `.typecheck.ts` (agent `build` runs `test:types`). Every branch above was mutation-checked. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- CHANGELOG.md | 4 +- docs/use-dkg/backpressure-observability.md | 2 +- packages/agent/src/dkg-agent-cg-resolve.ts | 181 +++++++++++------- packages/agent/src/dkg-agent-lifecycle.ts | 66 ++----- packages/agent/src/sync/catchup-policy.ts | 21 +- packages/agent/test/catchup-policy.test.ts | 32 ++++ .../test/catchup-retry-contract.typecheck.ts | 37 ++++ .../agent/test/cg-resolve-refresh.test.ts | 14 +- packages/agent/test/sync-policy.test.ts | 110 ++++++++--- .../cli/src/catchup-runner-worker-impl.ts | 31 +-- packages/cli/src/catchup-runner.ts | 63 ++++-- .../test/catchup-runner-worker-impl.test.ts | 174 ++++++++++++++++- packages/cli/test/catchup-runner.test.ts | 67 +++++++ .../context-graph-catchup-readiness.test.ts | 68 +++++++ 14 files changed, 682 insertions(+), 188 deletions(-) create mode 100644 packages/agent/test/catchup-retry-contract.typecheck.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 23e67dbb47..13392f15e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,8 +6,8 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Fixed -- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Metadata and unreachable peers are deliberately not treated as content, so a registered public graph that genuinely holds nothing still settles cleanly. -- **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s), and the timer is unreferenced so a pending backoff cannot outlive shutdown. +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. +- **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`. - **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. ### Changed diff --git a/docs/use-dkg/backpressure-observability.md b/docs/use-dkg/backpressure-observability.md index 03ab0d895c..1e4400826c 100644 --- a/docs/use-dkg/backpressure-observability.md +++ b/docs/use-dkg/backpressure-observability.md @@ -95,7 +95,7 @@ as `catchup-foreground` pressure. Both are read once at daemon start. | Variable | Default | Effect | | --- | --- | --- | | `DKG_CATCHUP_STOP_ON_PROOF` | on | The catch-up walks peers in escalating waves and stops once the resolved curator has settled every requested plane. Set to `0`, `false`, `no`, or `off` to restore the previous behaviour: every sync-capable peer, both requested planes, no early stop. Use this if a graph ever lands short — foreground catch-up optimises for one authoritative payload, while breadth remains the background reconcile lane's job. | -| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend waiting for local `sync-global` capacity before the job reports a retryable `deferred`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. | +| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend being **refused** by local `sync-global` admission before the job reports a retryable `deferred`. Measured from before the first attempt, so an attempt's own queue time counts against it. It does not cancel a round the scheduler has already accepted — that one is doing real work and is bounded by `SYNC_TOTAL_TIMEOUT_MS`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. | | `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Caps in-flight per-peer sync rounds, and therefore the widest escalation wave. Raising it above the `sync-global` queue depth lets a single catch-up saturate the scheduler against itself. | So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}` diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 9a0f8c33f6..70c3d7f3f6 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -522,6 +522,111 @@ async function applyContextGraphListPrivacy( .map(({ policyKnown: _policyKnown, ...row }) => row); } +/** Where a resolved catch-up sync peer came from; see {@link resolveCuratorSyncPeer}. */ +export interface SyncPeerResolution { + peerId?: string; + provenance: 'metadata' | 'bootstrap-hint' | 'none'; +} + +/** + * Resolve the curator peer for a Context Graph together with WHERE it came from. + * + * Two routes produce a peer id here and they are NOT interchangeable: + * + * - `'metadata'` — `/_meta` names a curator DID and it resolved to a peer. + * Authoritative: that peer speaks for the whole graph. + * - `'bootstrap-hint'` — the authenticated join-approval hint recorded in + * `preferredSyncPeers`, used while `_meta` has not arrived yet (and restored + * from the durable join-approved membership row after restart). It is a fine + * ranking signal but can be stale: peer ids are cryptographic identities, so + * a curator that rotated its libp2p key leaves an ordinary member sitting on + * the id the hint still names. + * + * Provenance is returned rather than inferred by a caller because the two + * routes routinely produce the SAME id — the join approval normally comes from + * the curator — so comparing the result against the hint cannot tell + * "metadata confirmed the curator" from "metadata found nothing and the hint + * was echoed back". Only the resolver knows which branch it took. + */ +export async function resolveCuratorSyncPeer( + agent: DKGAgent, + /** + * The agent's `preferredSyncPeers`, passed explicitly because it is both read + * and evicted here — and because that makes the resolver directly drivable in + * a test without standing up an agent. + */ + bootstrapHints: Map, + contextGraphId: string, + options: { signal?: AbortSignal } = {}, +): Promise { + const approvedCuratorPeerId = bootstrapHints.get(contextGraphId); + const fromHint = (): SyncPeerResolution => (approvedCuratorPeerId + ? { peerId: approvedCuratorPeerId, provenance: 'bootstrap-hint' } + : { provenance: 'none' }); + + const meta = await agent.getCgMeta(contextGraphId, { signal: options.signal }); + const curatorDid = meta.curator ?? meta.curators[0] ?? ''; + // Once `_meta` identifies a curator, that authoritative route must win over + // the bootstrap hint. + if (!curatorDid) return fromHint(); + const didPrefix = 'did:dkg:agent:'; + if (!curatorDid.startsWith(didPrefix)) return fromHint(); + const curatorIdentifier = curatorDid.slice(didPrefix.length); + + // Resolve curator identifier to a peer ID. The DID value is either a + // libp2p peer ID (legacy) or an Ethereum wallet address (V10). For + // wallet addresses, prefer the deterministic DKG_CREATOR triple (which + // stores the libp2p peer ID) over the agent registry (which may return + // an arbitrary match when multiple agents register the same wallet). + let curatorPeerId = curatorIdentifier; + if (curatorIdentifier.startsWith('0x')) { + let resolved = false; + + // Preferred: use the same projected metadata resolution as privacy and + // listing reads. AGENTS-only declarations can mark a graph private, so + // the refresh path must be able to discover their creator route too. + const creatorCandidates = [ + meta.creator, + ...meta.creators, + ].filter((value): value is string => Boolean(value)); + for (const creatorDid of creatorCandidates) { + if (creatorDid.startsWith(didPrefix)) { + const creatorId = creatorDid.slice(didPrefix.length); + if (!creatorId.startsWith('0x')) { + curatorPeerId = creatorId; + resolved = true; + break; + } + } + } + + // Fallback: agent registry lookup (non-deterministic if multiple agents + // share the same wallet address, but better than failing outright) + if (!resolved) { + try { + throwIfSyncAuthAborted(options.signal); + const agents = await agent.discovery.findAgents(); + throwIfSyncAuthAborted(options.signal); + const match = agents.find( + (a) => a.agentAddress?.toLowerCase() === curatorIdentifier.toLowerCase(), + ); + if (match) { + curatorPeerId = match.peerId; + resolved = true; + } + } catch { + throwIfSyncAuthAborted(options.signal); + /* registry unavailable */ + } + } + + if (!resolved) return fromHint(); + } + + bootstrapHints.delete(contextGraphId); + return { peerId: curatorPeerId, provenance: 'metadata' }; +} + export class ContextGraphResolveMethods extends DKGAgentBase { async getCgMeta( this: DKGAgent, @@ -1807,76 +1912,12 @@ export class ContextGraphResolveMethods extends DKGAgentBase { contextGraphId: string, options: { signal?: AbortSignal } = {}, ): Promise { - const approvedCuratorPeerId = this.preferredSyncPeers.get(contextGraphId); - const meta = await this.getCgMeta(contextGraphId, { signal: options.signal }); - const curatorDid = meta.curator ?? meta.curators[0] ?? ''; - if (!curatorDid) { - // Join approval authenticates the notification sender before recording - // this hint. It is the only curator route available during the bootstrap - // window where `_meta` has not arrived yet (and is restored from the - // durable join-approved membership row after restart). Once `_meta` - // identifies a curator, however, that authoritative route must win over - // the bootstrap hint. - return approvedCuratorPeerId; - } - const didPrefix = 'did:dkg:agent:'; - if (!curatorDid.startsWith(didPrefix)) { - return approvedCuratorPeerId; - } - const curatorIdentifier = curatorDid.slice(didPrefix.length); - - // Resolve curator identifier to a peer ID. The DID value is either a - // libp2p peer ID (legacy) or an Ethereum wallet address (V10). For - // wallet addresses, prefer the deterministic DKG_CREATOR triple (which - // stores the libp2p peer ID) over the agent registry (which may return - // an arbitrary match when multiple agents register the same wallet). - let curatorPeerId = curatorIdentifier; - if (curatorIdentifier.startsWith('0x')) { - let resolved = false; - - // Preferred: use the same projected metadata resolution as privacy and - // listing reads. AGENTS-only declarations can mark a graph private, so - // the refresh path must be able to discover their creator route too. - const creatorCandidates = [ - meta.creator, - ...meta.creators, - ].filter((value): value is string => Boolean(value)); - for (const creatorDid of creatorCandidates) { - if (creatorDid.startsWith(didPrefix)) { - const creatorId = creatorDid.slice(didPrefix.length); - if (!creatorId.startsWith('0x')) { - curatorPeerId = creatorId; - resolved = true; - break; - } - } - } - - // Fallback: agent registry lookup (non-deterministic if multiple agents - // share the same wallet address, but better than failing outright) - if (!resolved) { - try { - throwIfSyncAuthAborted(options.signal); - const agents = await this.discovery.findAgents(); - throwIfSyncAuthAborted(options.signal); - const match = agents.find( - (a) => a.agentAddress?.toLowerCase() === curatorIdentifier.toLowerCase(), - ); - if (match) { - curatorPeerId = match.peerId; - resolved = true; - } - } catch { - throwIfSyncAuthAborted(options.signal); - /* registry unavailable */ - } - } - - if (!resolved) return approvedCuratorPeerId; - } - - this.preferredSyncPeers.delete(contextGraphId); - return curatorPeerId; + return (await resolveCuratorSyncPeer( + this, + this.preferredSyncPeers, + contextGraphId, + options, + )).peerId; } async refreshMetaFromCurator( diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index dee073a4e8..f554cb41da 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -474,6 +474,7 @@ import { type SyncReconcilerProbe, type SyncReconcilerBackoff, } from './dkg-agent-types.js'; +import { resolveCuratorSyncPeer } from './dkg-agent-cg-resolve.js'; import { normalizePublishContextGraphId, isPublishAsyncQuadEnvelope, @@ -1160,28 +1161,6 @@ function emptySwmRecoveryResult(): RecoverContextGraphSwmResult { }; } -/** Where a resolved catch-up sync peer came from; see `resolveSyncPeerWithProvenance`. */ -export interface SyncPeerResolution { - peerId?: string; - provenance: 'metadata' | 'bootstrap-hint' | 'none'; -} - -/** - * Classify a resolved curator against the join-approval hint captured BEFORE - * resolution. Pure, so the distinction that decides whether one peer may stand - * for a whole Context Graph does not depend on any cache side effect. - */ -export function classifySyncPeerProvenance( - bootstrapHint: string | undefined, - curatorPeerId: string | undefined, -): SyncPeerResolution { - if (curatorPeerId && curatorPeerId !== bootstrapHint) { - return { peerId: curatorPeerId, provenance: 'metadata' }; - } - const peerId = curatorPeerId ?? bootstrapHint; - return peerId ? { peerId, provenance: 'bootstrap-hint' } : { provenance: 'none' }; -} - export class LifecycleSyncMethods extends DKGAgentBase { async runContextGraphSyncWithBackpressure(this: DKGAgent, ctx: OperationContext, @@ -6320,37 +6299,9 @@ export class LifecycleSyncMethods extends DKGAgentBase { return orderCatchupPeers(peers, preferredPeerId, privateOnly, this.knownCorePeerIds); } - /** - * Resolve the catch-up sync peer together with WHERE it came from. - * - * The distinction is load-bearing, so it is a return value rather than - * something a caller has to infer: only a metadata-resolved curator may let - * one peer's answer stand for the whole graph. The authenticated - * join-approval hint is a fine ranking signal but can be stale — peer ids are - * cryptographic identities, so a curator that has rotated its libp2p key - * leaves an ordinary member sitting on the id the hint still names. - * - * The bootstrap hint is captured BEFORE resolution so this does not depend on - * `resolveCuratorPeerId`'s cache-eviction side effect; the only property - * relied on is its documented contract, that it either returns a - * metadata-derived curator or echoes that same hint back. - */ - async resolveSyncPeerWithProvenance(this: DKGAgent, contextGraphId: string): Promise { - const bootstrapHint = this.preferredSyncPeers.get(contextGraphId); - return classifySyncPeerProvenance( - bootstrapHint, - await this.resolveCuratorPeerId(contextGraphId), - ); - } - async resolvePreferredSyncPeerId(this: DKGAgent, contextGraphId: string): Promise { - // Ranking takes the best peer available, whatever its provenance. Kept - // independent of the sibling method so this stays exercisable on its own. - const bootstrapHint = this.preferredSyncPeers.get(contextGraphId); - return classifySyncPeerProvenance( - bootstrapHint, - await this.resolveCuratorPeerId(contextGraphId), - ).peerId; + // Ranking takes the best peer available, whatever its provenance. + return (await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId)).peerId; } /** @@ -6360,13 +6311,20 @@ export class LifecycleSyncMethods extends DKGAgentBase { * {@link resolvePreferredSyncPeerId}. Only callers that let one peer's answer * stand for the whole graph — the foreground catch-up walk's early stop — * need this stricter notion, because a peer that happens to be ranked first - * must never be able to cut the walk short. + * must never be able to cut the walk short. The authenticated join-approval + * hint ranks but never settles: it can be stale, since a curator that rotated + * its libp2p key leaves an ordinary member sitting on the id it names. + * + * Provenance comes from {@link resolveCuratorSyncPeer} itself. Deriving it + * here — by comparing the resolved id against the hint — would be wrong in + * the ordinary case, where the join approval came from the curator and both + * routes name the SAME peer. */ async resolveAuthoritativeSyncPeerId( this: DKGAgent, contextGraphId: string, ): Promise { - const resolved = await this.resolveSyncPeerWithProvenance(contextGraphId); + const resolved = await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId); return resolved.provenance === 'metadata' ? resolved.peerId : undefined; } diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 0fd24c7d54..ae7484b835 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -153,6 +153,16 @@ export function nextCatchupBackpressureDelayMs(input: { * Cancellation needs no extra plumbing: an aborted admission raises an * `AbortError`, not a `SyncBackpressureBusyError`, so it never sets * `deferredBackpressure` and the loop's own guard exits on the next iteration. + * + * SCOPE. The budget bounds how long this plane keeps ASKING. It does not + * preempt a round the scheduler has already ACCEPTED: once admitted, the plane + * is doing (or waiting to do) real work, and the only cancellation seam reaching + * it — `operationSignal` — aborts the whole sync, so firing it at the deadline + * would kill productive catch-up under exactly the load this exists to survive. + * An accepted round is separately bounded by `SYNC_TOTAL_TIMEOUT_MS`, and the + * queue it waits in is depth-bounded, so the wait is finite either way. Refusal + * is the only signal that means "no capacity, come back later", and refusal is + * what this retries. */ export async function runCatchupPlaneWithPolicy( mode: CatchupMode, @@ -163,15 +173,18 @@ export async function runCatchupPlaneWithPolicy( priority: catchupPriorityForMode(mode), source: catchupSourceForMode(mode), }; - let result = await run(context); - if (mode !== 'foreground') return result; + if (mode !== 'foreground') return run(context); const now = options.now ?? Date.now; const wait = options.wait ?? defaultWait; const maxWaitMs = options.retry?.maxWaitMs ?? CATCHUP_BACKPRESSURE_MAX_WAIT_MS; - // Absolute deadline fixed once per plane, so retries cannot compound with the - // time the refused rounds themselves consumed. + // Absolute deadline fixed once per plane, taken BEFORE the first admission + // attempt. The attempts themselves are what consume the wall clock — a round + // that sits in the scheduler queue and then gets refused can take seconds — + // so starting the clock after the first one would make the budget "however + // long the first attempt took, PLUS maxWaitMs" instead of a per-plane bound. const retryUntil = now() + maxWaitMs; + let result = await run(context); for (let attempt = 0; ; attempt += 1) { if ((result.deferredBackpressure ?? 0) === 0) return result; const delayMs = nextCatchupBackpressureDelayMs({ diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index e863fcf991..a096efb59d 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -26,6 +26,10 @@ function virtualClock(startMs = 1_000) { waits.push(delayMs); nowMs += delayMs; }, + /** Charge time to something other than a backoff sleep — an attempt itself. */ + advance: (deltaMs: number) => { + nowMs += deltaMs; + }, elapsed: () => nowMs - startMs, }; } @@ -128,6 +132,34 @@ describe('runCatchupPlanesWithPolicy', () => { expect(clock.waits.reduce((sum, value) => sum + value, 0)).toBe(clock.elapsed()); }); + it('starts the budget before the first attempt, not after it', async () => { + // An attempt is not free: it can sit in the sync-global queue for seconds + // before being refused. Taking the deadline AFTER the first attempt made + // the real bound "however long that attempt took, PLUS maxWaitMs" — the one + // thing an operator setting a wall-clock budget is not asking for. + const clock = virtualClock(); + const attemptCostMs = 400; + const maxWaitMs = 1_000; + const syncDurable = vi.fn(async () => { + clock.advance(attemptCostMs); + return { deferredBackpressure: 1 }; + }); + const startedAt = clock.now(); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs }, + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + // At most ONE in-flight attempt may overrun the deadline — the policy + // cannot preempt a round it has already started. With the clock taken after + // the first attempt this lands at 1800 ms against a 1000 ms budget. + expect(clock.now() - startedAt).toBeLessThanOrEqual(maxWaitMs + attemptCostMs); + expect(syncDurable).toHaveBeenCalledTimes(2); + }); + it('never sleeps past the retry deadline', async () => { const clock = virtualClock(); const deadlineMs = 1_000; diff --git a/packages/agent/test/catchup-retry-contract.typecheck.ts b/packages/agent/test/catchup-retry-contract.typecheck.ts new file mode 100644 index 0000000000..314a8dd8d7 --- /dev/null +++ b/packages/agent/test/catchup-retry-contract.typecheck.ts @@ -0,0 +1,37 @@ +import { + runCatchupPlaneWithPolicy, + type CatchupPlanePolicyClock, + type CatchupPlanePolicyOptions, + type CatchupPlaneResult, +} from '@origintrail-official/dkg-agent'; + +// `retryDelaysMs` configured the fixed `[100, 250, 500]` ladder that issue #2006 +// replaced with a wall-clock budget. It is retained as `never` rather than +// deleted so that setting it is a COMPILE error instead of a silent no-op: an +// ignored `retryDelaysMs: [10]` would turn an intended 10 ms schedule into a +// wait of up to `CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, which is a far worse way to +// discover the change than a type error. +// +// That guarantee is a property of the PUBLISHED type, so it is pinned here +// rather than in a runtime test — no runtime assertion can observe it. Each +// `@ts-expect-error` below fails the build in BOTH directions: it errors today +// if the option were quietly made assignable again, and it errors as an unused +// suppression if the option were deleted outright (which would make a stale +// caller compile against a field that no longer exists at all). + +// @ts-expect-error retryDelaysMs was removed with the fixed ladder it configured. +const clock: CatchupPlanePolicyClock = { retryDelaysMs: [10, 20] }; + +const planes: CatchupPlanePolicyOptions = { + mode: 'foreground', + includeSharedMemory: false, + syncDurable: async () => ({}), + syncSharedMemory: async () => ({}), + // @ts-expect-error the same option is equally rejected on the two-plane options. + retryDelaysMs: [10], +}; + +// The replacement is `retry.maxWaitMs`, and it must stay assignable. +const supported: CatchupPlanePolicyClock = { retry: { maxWaitMs: 5_000 } }; + +export declare const pinned: [typeof clock, typeof planes, typeof supported, typeof runCatchupPlaneWithPolicy]; diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index 466d7d2a40..ac56b67a15 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1253,10 +1253,16 @@ describe('refreshMetaFromCurator', () => { expect(resolved).toBe(authoritativePeer); expect(preferredSyncPeers.has(contextGraphId)).toBe(false); - const lifecycleResolved = await LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId.call({ + // The same resolution through the lifecycle entry points, against the real + // metadata rather than a stubbed curator: the join-approved peer ranks only + // until `_meta` names someone, and the metadata answer is authoritative. + const lifecycleAgent = { + ...agent, preferredSyncPeers: new Map([[contextGraphId, bootstrapPeer]]), - resolveCuratorPeerId: async () => authoritativePeer, - } as never, contextGraphId); - expect(lifecycleResolved).toBe(authoritativePeer); + }; + expect(await LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId + .call(lifecycleAgent as never, contextGraphId)).toBe(authoritativePeer); + expect(await LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId + .call(lifecycleAgent as never, contextGraphId)).toBe(authoritativePeer); }); }); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 39bf98d0e7..53666ff634 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -9,7 +9,8 @@ import { syncPriorityClass, validateSyncResponderSnapshotLimitsConfig, } from '../src/sync/policy.js'; -import { classifySyncPeerProvenance } from '../src/dkg-agent-lifecycle.js'; +import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; +import { resolveCuratorSyncPeer } from '../src/dkg-agent-cg-resolve.js'; describe('sync Context Graph policy', () => { it('normalizes safe integer priorities and preserves stable input order for ties', () => { @@ -90,42 +91,101 @@ describe('normalizeSyncAdmissionSource', () => { }); }); -describe('classifySyncPeerProvenance', () => { +/** + * These drive the REAL resolver — `resolveCuratorSyncPeer` and the two + * lifecycle methods on their actual prototypes — not a stub of it. Which of + * the two routes produced a peer id is what decides whether one peer's answer + * may stand for a whole Context Graph, and that decision is made inside the + * resolver, so stubbing it out would leave the interesting half untested. + */ +describe('curator sync-peer provenance', () => { + const CG = 'cg/provenance'; const HINT = '12D3KooWBootstrapHint'; const CURATOR = '12D3KooWMetadataCurator'; - it('marks a metadata-resolved curator as authoritative', () => { - expect(classifySyncPeerProvenance(undefined, CURATOR)) - .toEqual({ peerId: CURATOR, provenance: 'metadata' }); - expect(classifySyncPeerProvenance(HINT, CURATOR)) + function agentWithMeta(meta: { + curator?: string; + curators?: string[]; + creator?: string; + creators?: string[]; + }, findAgents: () => Promise> = async () => []) { + return { + getCgMeta: async () => ({ curators: [], creators: [], ...meta }), + discovery: { findAgents }, + }; + } + + it('reports a metadata curator as authoritative EVEN when it equals the bootstrap hint', async () => { + // The ordinary case on a healthy network: the join approval came from the + // curator, so both routes name the same peer. Deriving provenance by + // comparing the resolved id against the hint therefore reads the normal + // case as "unconfirmed hint" and never lets the catch-up walk stop — + // exactly where the early-stop optimisation is worth the most. + const hints = new Map([[CG, CURATOR]]); + const agent = agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }); + + expect(await resolveCuratorSyncPeer(agent as never, hints, CG)) .toEqual({ peerId: CURATOR, provenance: 'metadata' }); + // …and the resolver consumed the hint now that metadata has confirmed it. + expect(hints.has(CG)).toBe(false); }); - it('marks an echoed bootstrap hint as NOT authoritative', () => { - // `resolveCuratorPeerId` echoes the join-approval hint when metadata - // resolves no curator. That hint can be stale — peer ids are cryptographic - // identities, so a curator that has rotated its libp2p key leaves an - // ordinary member on the id the hint still names — so it may rank the walk - // but must never let one peer stand for the whole graph. - expect(classifySyncPeerProvenance(HINT, HINT)) + it('marks an echoed bootstrap hint as NOT authoritative', async () => { + // With no curator in `_meta` the resolver echoes the join-approval hint. + // That hint can be stale — peer ids are cryptographic identities, so a + // curator that rotated its libp2p key leaves an ordinary member on the id + // it still names — so it may rank the walk but must never end it. + const hints = new Map([[CG, HINT]]); + + expect(await resolveCuratorSyncPeer(agentWithMeta({}) as never, hints, CG)) .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); - expect(classifySyncPeerProvenance(HINT, undefined)) + // A non-DKG curator DID is equally unresolvable. + expect(await resolveCuratorSyncPeer(agentWithMeta({ curator: 'did:web:example' }) as never, hints, CG)) .toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + // …as is a wallet-address curator no registry can resolve. + expect(await resolveCuratorSyncPeer( + agentWithMeta({ curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }) as never, + hints, + CG, + )).toEqual({ peerId: HINT, provenance: 'bootstrap-hint' }); + // The hint survives every fallback: it is still the best peer available. + expect(hints.get(CG)).toBe(HINT); }); - it('reports no peer when neither source produced one', () => { - expect(classifySyncPeerProvenance(undefined, undefined)) + it('reports no peer when neither route produced one', async () => { + expect(await resolveCuratorSyncPeer(agentWithMeta({}) as never, new Map(), CG)) .toEqual({ provenance: 'none' }); }); - it('keeps ranking availability identical to authority eligibility only for metadata', () => { - // The ranking caller takes `.peerId` regardless of provenance; the - // early-stop caller takes it only for 'metadata'. Pin that they differ - // exactly on the hint case. - for (const [hint, curator] of [[HINT, HINT], [HINT, undefined]] as const) { - const resolved = classifySyncPeerProvenance(hint, curator); - expect(resolved.peerId).toBe(HINT); - expect(resolved.provenance).not.toBe('metadata'); - } + it('resolves a wallet-address curator through the registry as authoritative', async () => { + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [{ agentAddress: '0x00000000000000000000000000000000000000AB', peerId: CURATOR }], + ); + + expect(await resolveCuratorSyncPeer(agent as never, hints, CG)) + .toEqual({ peerId: CURATOR, provenance: 'metadata' }); + }); + + it('ranks on any provenance but only lets metadata settle the walk', async () => { + // The two lifecycle entry points, on their real prototypes: ranking takes + // whatever peer is available, authority takes it only from metadata. + const confirmedCurator = { + preferredSyncPeers: new Map([[CG, CURATOR]]), + ...agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }), + }; + const hintOnly = { + preferredSyncPeers: new Map([[CG, HINT]]), + ...agentWithMeta({}), + }; + const rank = LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId; + const authority = LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId; + + expect(await rank.call(confirmedCurator as never, CG)).toBe(CURATOR); + expect(await authority.call(confirmedCurator as never, CG)).toBe(CURATOR); + + expect(await rank.call(hintOnly as never, CG)).toBe(HINT); + expect(await authority.call(hintOnly as never, CG)).toBeUndefined(); }); }); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 535cb1cf30..a563843e1d 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -18,6 +18,7 @@ import { catchupPeerSucceeded, catchupPlaneProvenByData, type CatchupJobResult, + type CatchupPlaneCompletionEvidence, type CatchupRunRequest, } from './catchup-runner.js'; @@ -128,8 +129,13 @@ async function runCatchup(request: CatchupRunRequest): Promise let noProtocolPeers = 0; const cleanPlaneCompletions: NonNullable = { - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0 }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, }; const diagnostics: NonNullable = { @@ -211,19 +217,19 @@ async function runCatchup(request: CatchupRunRequest): Promise /** * Whether the curator's round settles a plane well enough to stop walking. * - * Verified content always does. A clean EMPTY round only does for a PUBLIC + * Verified content always does. A content-free round only does for a PUBLIC * graph: readiness deliberately refuses to prove a private plane from an * empty response, so stopping on one would strand the walk without proving * anything — skipping fallback peers that may hold authorized private data * and turning a recoverable catch-up into `unreachable`. A verified * private-only response is content, not emptiness, and still counts. + * + * `authorityEmptyPeers` is set by the same reducer readiness consumes, so the + * stop condition and the readiness verdict cannot drift apart. */ - const authoritySettles = (evidence: { - verifiedDataPeers: number; - verifiedPrivateOnlyPeers?: number; - emptyPeers: number; - }): boolean => catchupPlaneProvenByData(evidence) - || (!prepared.isPrivateContextGraph && evidence.emptyPeers > 0); + const authoritySettles = (evidence: CatchupPlaneCompletionEvidence): boolean => + catchupPlaneProvenByData(evidence) + || (!prepared.isPrivateContextGraph && (evidence.authorityEmptyPeers ?? 0) > 0); // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we // can from the other peers instead of failing the entire subscribe/catch-up. @@ -298,7 +304,10 @@ async function runCatchup(request: CatchupRunRequest): Promise (diagnostics.durable.deniedPhases ?? 0) + (durable.deniedPhases ?? 0); peerDenied = peerDenied || durable.deniedPhases > 0; - const durableEvidence = catchupPeerPlaneEvidence(durable, { complete: durable.complete }); + const durableEvidence = catchupPeerPlaneEvidence(durable, { + complete: durable.complete, + fromAuthority, + }); addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); // The curator answering cleanly settles this plane whether it carried // data or was legitimately empty: "the host says there is nothing here" @@ -336,7 +345,7 @@ async function runCatchup(request: CatchupRunRequest): Promise // Shared memory carries no verified-private-only signal, so the shared // evidence only ever has data/empty set — the same reducer still applies. - const sharedEvidence = catchupPeerPlaneEvidence(shared); + const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority }); addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); // Same rule as durable: the curator settles the plane by answering // cleanly, with data or empty. Shared memory is frequently empty for a diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index e478be9422..0566dc2f4e 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -57,16 +57,9 @@ export interface CatchupJobResult { * the same plane cleanly and stored verified data. */ cleanPlaneCompletions?: { - durable: { - verifiedDataPeers: number; - /** Peers that cleanly verified one or more V2 KAs with no public triples. */ - verifiedPrivateOnlyPeers: number; - emptyPeers: number; - }; - sharedMemory: { - verifiedDataPeers: number; - emptyPeers: number; - }; + /** Always carries `verifiedPrivateOnlyPeers`; only the durable plane can produce it. */ + durable: CatchupPlaneCompletionEvidence & { verifiedPrivateOnlyPeers: number }; + sharedMemory: CatchupPlaneCompletionEvidence; }; diagnostics?: { noProtocolPeers: number; @@ -517,6 +510,12 @@ export interface CatchupPlaneCompletionEvidence { /** Peers that cleanly verified one or more V2 KAs with no public triples. */ verifiedPrivateOnlyPeers?: number; emptyPeers: number; + /** + * The metadata-resolved curator cleanly completed this plane while hosting + * the graph and carrying no data at all. See + * {@link catchupPlaneProvenByUnanimousEmpty}. + */ + authorityEmptyPeers?: number; } /** The aggregate per-plane counters a whole-round verdict is allowed to consult. */ @@ -543,15 +542,34 @@ export interface CatchupPlaneRoundDiagnostics { * A plane that did not complete cleanly contributes nothing at all. */ export function catchupPeerPlaneEvidence( - plane: (CatchupPhaseProgress & { emptyResponses?: number }) | null | undefined, - options: { complete?: boolean } = {}, + plane: + | (CatchupPhaseProgress & { emptyResponses?: number; fetchedDataTriples?: number }) + | null + | undefined, + options: { complete?: boolean; fromAuthority?: boolean } = {}, ): CatchupPlaneCompletionEvidence { - const none = { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }; + const none = { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + }; if (!plane || !catchupPlaneCompletedWithoutFailure(plane, options.complete)) return none; + // "The host says there is nothing here." Only the curator can say it: a + // response is content-free either by being wire-empty or by carrying nothing + // but `_meta`, and only the metadata-resolved curator's silence about data + // means the graph has none. Any other peer's identical answer just means that + // peer does not have it. + const carriedNoData = (plane.insertedDataTriples ?? 0) === 0 + && (plane.fetchedDataTriples ?? 0) === 0; + const answered = (plane.emptyResponses ?? 0) > 0 + || (plane.metaOnlyResponses ?? 0) > 0 + || (plane.insertedMetaTriples ?? 0) > 0; return { verifiedDataPeers: (plane.insertedDataTriples ?? 0) > 0 ? 1 : 0, verifiedPrivateOnlyPeers: (plane.verifiedPrivateOnlyResponses ?? 0) > 0 ? 1 : 0, emptyPeers: (plane.emptyResponses ?? 0) > 0 ? 1 : 0, + authorityEmptyPeers: options.fromAuthority && carriedNoData && answered ? 1 : 0, }; } @@ -566,6 +584,9 @@ export function addCatchupPlaneEvidence( + peer.verifiedPrivateOnlyPeers; } total.emptyPeers += peer.emptyPeers; + if (peer.authorityEmptyPeers) { + total.authorityEmptyPeers = (total.authorityEmptyPeers ?? 0) + peer.authorityEmptyPeers; + } } /** @@ -598,6 +619,15 @@ export function catchupPlaneProvenByData( * settled issue #2006's run as `done` with 1 KA out of 40, and either clause * kills it on its own. * + * A registered public graph that really is empty is therefore proven by its + * CURATOR instead — `authorityEmptyPeers`. Such a graph still carries + * definition triples in its own `/_meta`, so the peer hosting it answers + * metadata-only rather than wire-empty and could never satisfy the round rule + * above. Only the metadata-resolved curator counts: any OTHER peer's + * metadata-only round is the commonest state on the network — a member that + * has `_meta` but has not synced the data yet — and accepting it would resettle + * issue #2006's exact failure as `done` with zero Knowledge Assets. + * * Two counters are deliberately NOT consulted: * * - `fetchedMetaTriples`. Every registered Context Graph carries definition @@ -624,6 +654,13 @@ export function catchupPlaneProvenByUnanimousEmpty( // private graph is fully synchronized; that stays unchanged. if (options.isPrivate) return false; if (catchupPlaneProvenByData(completion)) return false; + // The curator hosting the graph and carrying no data settles it on its own — + // it is the reference for the whole graph, so another peer failing part-way + // cannot contradict it. Another peer DELIVERING data can, which is the one + // guard kept: that means the curator's view is behind the network's. + if ((completion?.authorityEmptyPeers ?? 0) > 0) { + return (diagnostics?.fetchedDataTriples ?? 0) === 0; + } const cleanEmptyObserved = (completion?.emptyPeers ?? 0) > 0 || (diagnostics?.emptyResponses ?? 0) > 0; if (!cleanEmptyObserved) return false; diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 9bee90395b..a351df3ff8 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -563,6 +563,158 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.cleanPlaneCompletions?.durable.verifiedPrivateOnlyPeers).toBe(1); }); + it('does not let an empty curator round settle a PRIVATE shared-memory plane', async () => { + // The shared-memory half of the private rule. It needs its own coverage: + // `includeSharedMemory` defaults to true on subscribe and shared memory is + // frequently empty, so this is the plane an over-eager empty rule would + // settle first — stranding the walk before any authorized peer holding SWM + // data is contacted. The durable plane settles by verified content here, so + // only the shared plane's rule is under test. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-private-swm', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: true, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 8, + fetchedMetaTriples: 8, + fetchedDataTriples: 0, + insertedMetaTriples: 8, + insertedDataTriples: 0, + verifiedPrivateOnlyResponses: 1, + }; + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 1, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // Durable settled on the curator's verified content and is not re-pulled… + expect(durableCalls).toEqual(['peer-0']); + // …while the unproven shared plane keeps walking every remaining peer. + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(1); + }); + + it('settles a public plane when the CURATOR hosts the graph and has no data', async () => { + // A registered public Context Graph with no Knowledge Assets yet. Its host + // still serves the CG definition triples from `/_meta`, so it answers + // metadata-only — never wire-empty — and no whole-round emptiness rule can + // fire for it. The curator saying "I host this and there is nothing in it" + // is the only evidence that exists, and without it such a graph would sit + // at `unreachable` forever while re-walking every peer on every retry. + const peerIds = Array.from({ length: 8 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-registered-empty', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(durableCalls).toEqual(['peer-0']); + expect(result.peersNotAttempted).toBe(peerIds.length - 1); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(1); + // The same round from a peer that is NOT the curator proves nothing: it is + // what any member holding `_meta` but no data looks like. + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBe(0); + }); + + it('does not let a non-curator metadata-only round stop the walk', async () => { + // The counterpart of the test above, and the reason it is scoped to the + // curator: mid-sync members answering metadata-only are the commonest + // state on the network. Accepting theirs would resettle #2006 exactly — + // `done` with zero Knowledge Assets out of forty. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-members-only', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...durableCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(0); + }); + it('does not let a bootstrap-hint preferred peer stop the walk', async () => { // `resolvePreferredSyncPeerId` falls back to the authenticated join-approval // hint when metadata resolves no curator. That hint can be stale — a curator @@ -996,8 +1148,13 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.diagnostics?.durable.deniedPhases).toBe(1); expect(result.diagnostics?.sharedMemory.deniedPhases).toBe(1); expect(result.cleanPlaneCompletions).toEqual({ - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0 }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: 0, + authorityEmptyPeers: 0, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, }); expect(result.diagnostics?.durable.verifiedPrivateOnlyResponses).toBe(0); }); @@ -1046,6 +1203,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 1, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, }); }); @@ -1084,6 +1242,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, }); }); @@ -1129,6 +1288,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 1, emptyPeers: 0, + authorityEmptyPeers: 0, }); }); @@ -1169,6 +1329,7 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, + authorityEmptyPeers: 0, }); }); @@ -1227,8 +1388,13 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) sharedMemorySynced: 0, }); expect(result.cleanPlaneCompletions).toEqual({ - durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: peerIds.length }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: peerIds.length }, + durable: { + verifiedDataPeers: 0, + verifiedPrivateOnlyPeers: 0, + emptyPeers: peerIds.length, + authorityEmptyPeers: 0, + }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: peerIds.length, authorityEmptyPeers: 0 }, }); }); }); diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 0189378ad5..3fbedf3445 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it, vi } from 'vitest'; import { + catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, @@ -741,6 +742,72 @@ describe('catch-up plane proof predicates', () => { )).toBe(false); }); + // A registered public graph that really is empty still carries definition + // triples in its own `/_meta`, so the peer hosting it answers + // metadata-only, never wire-empty. Nothing in the whole-round rule above can + // ever fire for it — the curator has to say so itself. + describe('an empty graph whose only responder is its curator', () => { + const hostedEmptyRound = { + insertedTriples: 9, + insertedMetaTriples: 9, + insertedDataTriples: 0, + fetchedDataTriples: 0, + metaOnlyResponses: 1, + emptyResponses: 0, + completedPhases: 2, + }; + const hostedEmptyDiagnostics = { + ...cleanEmptyRound, + fetchedMetaTriples: 9, + emptyResponses: 0, + }; + + it('counts the curator, and ONLY the curator, as hosted-empty evidence', () => { + expect(catchupPeerPlaneEvidence(hostedEmptyRound, { + complete: true, + fromAuthority: true, + })).toMatchObject({ verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 1 }); + // The identical round from any other peer is the commonest state on the + // network — a member holding `_meta` that has not synced the data yet — + // and counting it would resettle #2006 as `done` with zero KAs. + expect(catchupPeerPlaneEvidence(hostedEmptyRound, { complete: true })) + .toMatchObject({ authorityEmptyPeers: 0 }); + // Neither does a curator round that fetched data but inserted none. + expect(catchupPeerPlaneEvidence( + { ...hostedEmptyRound, fetchedDataTriples: 4_000 }, + { complete: true, fromAuthority: true }, + )).toMatchObject({ authorityEmptyPeers: 0 }); + }); + + it('proves the public plane with no wire-empty response anywhere in the round', () => { + const completion = { ...noEvidence, authorityEmptyPeers: 1 }; + expect(catchupPlaneProvenByUnanimousEmpty( + completion, + hostedEmptyDiagnostics, + { isPrivate: false }, + )).toBe(true); + expect(catchupPlaneReady(completion, hostedEmptyDiagnostics, { isPrivate: false })).toBe(true); + // Without the curator's own evidence the same round proves nothing. + expect(catchupPlaneReady(noEvidence, hostedEmptyDiagnostics, { isPrivate: false })).toBe(false); + }); + + it('is voided when another peer delivered data the curator did not have', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + { ...hostedEmptyDiagnostics, fetchedDataTriples: 122_705 }, + { isPrivate: false }, + )).toBe(false); + }); + + it('never proves a private plane', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + hostedEmptyDiagnostics, + { isPrivate: true }, + )).toBe(false); + }); + }); + it('accepts either evidence carrier for the clean empty completion', () => { // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter // (`diagnostics.emptyResponses`) are separate carriers, and the legacy diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 64edd4a5b0..6ac48c89f4 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -321,4 +321,72 @@ describe('context graph catch-up readiness classification', () => { expect(classification.jobStatus).toBe('unreachable'); expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); }); + + // A registered public graph with no Knowledge Assets yet. Its host serves the + // CG definition triples from `/_meta`, so it answers metadata-only rather + // than wire-empty and the whole-round rule above can never fire — no peer in + // the round produced an `emptyResponses`. The curator's own hosted-empty + // round is the only evidence such a graph can produce. + function curatorHostedEmptyResult(): CatchupJobResult { + const result = publicEmptyRoundResult(); + if (!result.cleanPlaneCompletions || !result.diagnostics?.durable) { + throw new Error('durable completion evidence missing'); + } + result.cleanPlaneCompletions.durable.emptyPeers = 0; + result.cleanPlaneCompletions.durable.authorityEmptyPeers = 1; + result.diagnostics.durable.emptyResponses = 0; + result.diagnostics.durable.metaOnlyResponses = 1; + result.diagnostics.durable.fetchedMetaTriples = 9; + result.diagnostics.durable.insertedMetaTriples = 9; + return result; + } + + it('settles a registered-but-empty public graph on the curator hosted-empty round', () => { + expect(classifyContextGraphCatchupReadiness({ + result: curatorHostedEmptyResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + statePatch: { synced: true }, + readinessPatch: { durableVerified: true }, + }); + }); + + it.each([ + ['the round came from members rather than the curator', (result: CatchupJobResult) => { + result.cleanPlaneCompletions!.durable.authorityEmptyPeers = 0; + }], + ['another peer delivered data the curator did not have', (result: CatchupJobResult) => { + result.diagnostics!.durable.fetchedDataTriples = 122_705; + }], + ])('keeps the same round unready when %s', (_label, mutate) => { + const result = curatorHostedEmptyResult(); + mutate(result); + + expect(classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'unreachable', + readinessPatch: { durableVerified: false }, + }); + }); + + it('never settles a PRIVATE plane on a curator hosted-empty round', () => { + // Private planes stay proof-by-content only: an authorized-but-filtered + // response is indistinguishable from an empty one on this side of the wire. + expect(classifyContextGraphCatchupReadiness({ + result: curatorHostedEmptyResult(), + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: true, + readinessBeforeCatchup, + }).jobStatus).toBe('unreachable'); + }); }); From fb0db7d318813c38e3d9aeba2dfad204fa3055ba Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:01:44 +0200 Subject: [PATCH 18/44] fix(sync): integrity rejections void an empty round; one boundary resolution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 9. An integrity rejection is not "a peer that failed" — it is a peer that SERVED CONTENT for this Context Graph which then failed verification, so it is positive evidence the graph is NOT empty. `classifyDurableProgress` already treats `rejectedKcs` / `dataRejectedMissingMeta` as blocking failures per peer, but the whole-round empty rule did not consult them, so one stranger's clean-empty answer could settle a plane whose only real content had been rejected. They now void the verdict ahead of even the curator's own word — unlike a plain transport or phase failure, which the curator's answer does outrank. `prepareCatchup` resolved the sync peer TWICE, once per notion. Each resolution reads `/_meta` and can drive the agent-registry fallback, and the resolver evicts the bootstrap hint once metadata confirms a curator — so the second call was neither free nor the same call. It is one `resolveSyncPeerWithProvenance` now, with `authoritativeSyncPeerId` as the single definition of which peer may end the walk. The narrow wrappers deliberately do NOT route through the sibling method: several suites invoke them on hand-built receivers, which a `this` hop would break. The `retryDelaysMs` type test gave false confidence and I had claimed otherwise in a review reply. Excess-property checking rejects an object literal whether the member is `never` or ABSENT, so a literal-only test passes in both worlds. Verified by deleting the member from the built declaration: `test:types` stayed green. It is now pinned two ways deletion breaks — an indexed access on the member, and a stale options VARIABLE, where excess properties are permitted and only a declared `never` can refuse them. Both mutation directions now fail. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-cg-resolve.ts | 9 ++++ packages/agent/src/dkg-agent-lifecycle.ts | 46 ++++++++++++++----- .../test/catchup-retry-contract.typecheck.ts | 38 ++++++++++++--- packages/agent/test/sync-policy.test.ts | 31 ++++++++++++- packages/cli/src/catchup-runner.ts | 39 ++++++++++++---- .../catchup-runner-worker-lifecycle.test.ts | 43 ++++++++++++++--- packages/cli/test/catchup-runner.test.ts | 20 ++++++++ 7 files changed, 190 insertions(+), 36 deletions(-) diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 70c3d7f3f6..f670ee2b52 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -528,6 +528,15 @@ export interface SyncPeerResolution { provenance: 'metadata' | 'bootstrap-hint' | 'none'; } +/** + * The peer allowed to let one answer stand for a whole Context Graph — a + * metadata-resolved curator and nothing else. A single definition so the walk's + * early-stop rule cannot be restated slightly differently at another call site. + */ +export function authoritativeSyncPeerId(resolution: SyncPeerResolution): string | undefined { + return resolution.provenance === 'metadata' ? resolution.peerId : undefined; +} + /** * Resolve the curator peer for a Context Graph together with WHERE it came from. * diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index f554cb41da..0247123adf 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -474,7 +474,11 @@ import { type SyncReconcilerProbe, type SyncReconcilerBackoff, } from './dkg-agent-types.js'; -import { resolveCuratorSyncPeer } from './dkg-agent-cg-resolve.js'; +import { + authoritativeSyncPeerId, + resolveCuratorSyncPeer, + type SyncPeerResolution, +} from './dkg-agent-cg-resolve.js'; import { normalizePublishContextGraphId, isPublishAsyncQuadEnvelope, @@ -6299,22 +6303,39 @@ export class LifecycleSyncMethods extends DKGAgentBase { return orderCatchupPeers(peers, preferredPeerId, privateOnly, this.knownCorePeerIds); } + /** + * Resolve the catch-up sync peer ONCE, with both notions the walk needs. + * + * They are one resolution, not two: ranking takes the best peer available + * whatever its provenance, while letting one peer's answer stand for the + * whole graph requires a metadata-resolved curator. The authenticated + * join-approval hint ranks but never settles — it can be stale, since a + * curator that rotated its libp2p key leaves an ordinary member sitting on + * the id it names. + * + * Deriving that distinction from two calls would read `_meta` twice (and run + * the registry fallback twice for a wallet-address curator) per catch-up, and + * would hide that the resolver has a side effect — it evicts the bootstrap + * hint once metadata confirms a curator, so the second call is not the same + * call as the first. + */ + async resolveSyncPeerWithProvenance( + this: DKGAgent, + contextGraphId: string, + ): Promise { + return resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId); + } + async resolvePreferredSyncPeerId(this: DKGAgent, contextGraphId: string): Promise { - // Ranking takes the best peer available, whatever its provenance. + // Deliberately NOT routed through the sibling method: each of these is one + // resolution on its own, and going through `this` would make them + // unusable against the hand-built receivers several suites call them on. return (await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId)).peerId; } /** * The sync peer ONLY when it is a metadata-resolved curator. * - * Callers that merely want to try the best peer first should use - * {@link resolvePreferredSyncPeerId}. Only callers that let one peer's answer - * stand for the whole graph — the foreground catch-up walk's early stop — - * need this stricter notion, because a peer that happens to be ranked first - * must never be able to cut the walk short. The authenticated join-approval - * hint ranks but never settles: it can be stale, since a curator that rotated - * its libp2p key leaves an ordinary member sitting on the id it names. - * * Provenance comes from {@link resolveCuratorSyncPeer} itself. Deriving it * here — by comparing the resolved id against the hint — would be wrong in * the ordinary case, where the join approval came from the curator and both @@ -6324,8 +6345,9 @@ export class LifecycleSyncMethods extends DKGAgentBase { this: DKGAgent, contextGraphId: string, ): Promise { - const resolved = await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId); - return resolved.provenance === 'metadata' ? resolved.peerId : undefined; + return authoritativeSyncPeerId( + await resolveCuratorSyncPeer(this, this.preferredSyncPeers, contextGraphId), + ); } async ensurePeerConnected(this: DKGAgent, peerId: string): Promise { diff --git a/packages/agent/test/catchup-retry-contract.typecheck.ts b/packages/agent/test/catchup-retry-contract.typecheck.ts index 314a8dd8d7..1db4e2cfec 100644 --- a/packages/agent/test/catchup-retry-contract.typecheck.ts +++ b/packages/agent/test/catchup-retry-contract.typecheck.ts @@ -13,11 +13,30 @@ import { // discover the change than a type error. // // That guarantee is a property of the PUBLISHED type, so it is pinned here -// rather than in a runtime test — no runtime assertion can observe it. Each -// `@ts-expect-error` below fails the build in BOTH directions: it errors today -// if the option were quietly made assignable again, and it errors as an unused -// suppression if the option were deleted outright (which would make a stale -// caller compile against a field that no longer exists at all). +// rather than in a runtime test — no runtime assertion can observe it. +// +// Object literals alone CANNOT carry it. Excess-property checking rejects +// `{ retryDelaysMs: [...] }` against an annotated target whether the member is +// declared `never` or absent entirely, so a literal-only test passes in both +// worlds and proves nothing about the difference. Deleting the member is +// precisely the stale-caller silent-ignore case the `never` exists to prevent, +// so it is pinned two ways that a deletion breaks: an indexed access on the +// member itself, and a stale options object flowing through a VARIABLE, where +// excess properties are permitted and only a declared `never` can refuse them. + +// Fails to compile (TS2339) if the member is deleted rather than kept `never`. +declare const removedLadder: CatchupPlanePolicyClock['retryDelaysMs']; +// …and `undefined` is the only value it can hold. +const ladderIsUninhabited: undefined = removedLadder; + +declare const staleCallerOptions: { + retry: { maxWaitMs: number }; + retryDelaysMs: number[]; +}; +// @ts-expect-error a stale options VARIABLE carrying the removed ladder must not +// flow in structurally — this is the case excess-property checking would let by, +// and the one that silently reverted to a full-budget wait before the `never`. +const stale: CatchupPlanePolicyClock = staleCallerOptions; // @ts-expect-error retryDelaysMs was removed with the fixed ladder it configured. const clock: CatchupPlanePolicyClock = { retryDelaysMs: [10, 20] }; @@ -34,4 +53,11 @@ const planes: CatchupPlanePolicyOptions // The replacement is `retry.maxWaitMs`, and it must stay assignable. const supported: CatchupPlanePolicyClock = { retry: { maxWaitMs: 5_000 } }; -export declare const pinned: [typeof clock, typeof planes, typeof supported, typeof runCatchupPlaneWithPolicy]; +export declare const pinned: [ + typeof clock, + typeof planes, + typeof supported, + typeof stale, + typeof ladderIsUninhabited, + typeof runCatchupPlaneWithPolicy, +]; diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 53666ff634..f55d0fb696 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -10,7 +10,7 @@ import { validateSyncResponderSnapshotLimitsConfig, } from '../src/sync/policy.js'; import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; -import { resolveCuratorSyncPeer } from '../src/dkg-agent-cg-resolve.js'; +import { authoritativeSyncPeerId, resolveCuratorSyncPeer } from '../src/dkg-agent-cg-resolve.js'; describe('sync Context Graph policy', () => { it('normalizes safe integer priorities and preserves stable input order for ties', () => { @@ -168,6 +168,35 @@ describe('curator sync-peer provenance', () => { .toEqual({ peerId: CURATOR, provenance: 'metadata' }); }); + it('answers ranking and authority from ONE resolution', async () => { + // The catch-up boundary needs both notions, and resolving twice is not + // free or even equivalent: each resolution reads `_meta` (and can drive the + // registry fallback), and the resolver EVICTS the bootstrap hint once + // metadata confirms a curator, so the second call runs against a different + // map than the first. + let metaReads = 0; + const agent = { + preferredSyncPeers: new Map([[CG, CURATOR]]), + getCgMeta: async () => { + metaReads += 1; + return { curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }; + }, + discovery: { findAgents: async () => [] }, + }; + + const resolved = await LifecycleSyncMethods.prototype.resolveSyncPeerWithProvenance + .call(agent as never, CG); + + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'metadata' }); + expect(metaReads).toBe(1); + // Both narrow notions are derivable from it, matching the wrappers exactly. + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBe(CURATOR); + expect(authoritativeSyncPeerId({ peerId: CURATOR, provenance: 'bootstrap-hint' })) + .toBeUndefined(); + expect(authoritativeSyncPeerId({ provenance: 'none' })).toBeUndefined(); + }); + it('ranks on any provenance but only lets metadata settle the walk', async () => { // The two lifecycle entry points, on their real prototypes: ranking takes // whatever peer is available, authority takes it only from metadata. diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 0566dc2f4e..f4f3445c3e 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -528,6 +528,9 @@ export interface CatchupPlaneRoundDiagnostics { timedOutPhases?: number; deniedPhases?: number; deferredBackpressure?: number; + /** Durable-only integrity rejections; the shared-memory plane never sets them. */ + dataRejectedMissingMeta?: number; + rejectedKcs?: number; } /** @@ -654,10 +657,18 @@ export function catchupPlaneProvenByUnanimousEmpty( // private graph is fully synchronized; that stays unchanged. if (options.isPrivate) return false; if (catchupPlaneProvenByData(completion)) return false; + // An integrity rejection is not "a peer that failed" — it is a peer that + // SERVED CONTENT for this graph which then failed verification. That is + // positive evidence the graph is not empty, so it voids an empty verdict + // outright, ahead of even the curator's own word. `classifyDurableProgress` + // already treats these as blocking failures per peer; this is the same rule + // applied to the round. + if ((diagnostics?.dataRejectedMissingMeta ?? 0) > 0 + || (diagnostics?.rejectedKcs ?? 0) > 0) return false; // The curator hosting the graph and carrying no data settles it on its own — - // it is the reference for the whole graph, so another peer failing part-way - // cannot contradict it. Another peer DELIVERING data can, which is the one - // guard kept: that means the curator's view is behind the network's. + // it is the reference for the whole graph, so another peer merely failing + // part-way cannot contradict it. Another peer DELIVERING data can, which is + // the guard kept here: that means the curator's view is behind the network's. if ((completion?.authorityEmptyPeers ?? 0) > 0) { return (diagnostics?.fetchedDataTriples ?? 0) === 0; } @@ -858,13 +869,21 @@ class WorkerCatchupRunner implements CatchupRunner { case 'prepareCatchup': { const [contextGraphId] = args as [string]; const isPrivateContextGraph = await agent.isPrivateContextGraph(contextGraphId); - const preferredPeerId = await agent.resolvePreferredSyncPeerId(contextGraphId); - // Ranking uses the preferred peer; letting ONE peer's answer stand for - // the whole graph requires the stricter notion. A join-approval - // bootstrap hint is authenticated but can be stale, so it orders the - // walk without being allowed to end it. - const authoritativePeerId = typeof agent.resolveAuthoritativeSyncPeerId === 'function' - ? await agent.resolveAuthoritativeSyncPeerId(contextGraphId) + // ONE resolution, two notions. Ranking uses whatever peer is available; + // letting one peer's answer stand for the whole graph requires the + // stricter notion, because a join-approval bootstrap hint is + // authenticated but can be stale — it orders the walk without being + // allowed to end it. Resolving twice would read `_meta` twice (and run + // the registry fallback twice for a wallet-address curator) per + // catch-up, and the resolver evicts the bootstrap hint once metadata + // confirms a curator, so the second call is not the same call. + const resolution: { peerId?: string; provenance?: string } = + typeof agent.resolveSyncPeerWithProvenance === 'function' + ? await agent.resolveSyncPeerWithProvenance(contextGraphId) + : { peerId: await agent.resolvePreferredSyncPeerId(contextGraphId) }; + const preferredPeerId: string | undefined = resolution.peerId; + const authoritativePeerId = resolution.provenance === 'metadata' + ? resolution.peerId : undefined; if (preferredPeerId) { await agent.ensurePeerConnected(preferredPeerId); diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index 020e0374f1..2463616efe 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -131,10 +131,13 @@ describe('WorkerCatchupRunner lifecycle', () => { describe('WorkerCatchupRunner agent bridge', () => { function bridgeAgent(overrides: Record = {}) { const calls: Record = { durable: [], shared: [] }; + let resolutionCalls = 0; const agent = { isPrivateContextGraph: async () => false, - resolvePreferredSyncPeerId: async () => 'peer-hint', - resolveAuthoritativeSyncPeerId: async () => undefined, + resolveSyncPeerWithProvenance: async () => { + resolutionCalls += 1; + return { peerId: 'peer-hint', provenance: 'bootstrap-hint' }; + }, ensurePeerConnected: async () => {}, primeCatchupConnections: async () => {}, selectCatchupPeers: (peers: Array<{ toString(): string }>) => peers, @@ -149,7 +152,11 @@ describe('WorkerCatchupRunner agent bridge', () => { }, ...overrides, }; - return { agent: agent as unknown as DKGAgent, calls }; + return { + agent: agent as unknown as DKGAgent, + calls, + resolutionCalls: () => resolutionCalls, + }; } /** Drive one `invoke` through the real bridge and return what it posted back. */ @@ -164,24 +171,46 @@ describe('WorkerCatchupRunner agent bridge', () => { } it('does not report a bootstrap-hint peer as the catch-up authority', async () => { - const { agent } = bridgeAgent(); + const { agent, resolutionCalls } = bridgeAgent(); const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-hint']); // The hint still ranks the walk… expect(posted.result.preferredPeerId).toBe('peer-hint'); // …but must not be handed to the worker as an authority. expect(posted.result.authoritativePeerId).toBeUndefined(); + // …and both notions came from ONE resolution. The resolver reads `_meta` + // (and may hit the agent registry), and it evicts the bootstrap hint once + // metadata confirms a curator — so a second call is neither free nor the + // same call. + expect(resolutionCalls()).toBe(1); }); it('reports a metadata-resolved curator as the catch-up authority', async () => { - const { agent } = bridgeAgent({ - resolvePreferredSyncPeerId: async () => 'peer-curator', - resolveAuthoritativeSyncPeerId: async () => 'peer-curator', + const { agent, resolutionCalls } = bridgeAgent({ + resolveSyncPeerWithProvenance: async () => ({ + peerId: 'peer-curator', + provenance: 'metadata', + }), }); const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-meta']); expect(posted.result.preferredPeerId).toBe('peer-curator'); expect(posted.result.authoritativePeerId).toBe('peer-curator'); + expect(resolutionCalls()).toBe(0); + }); + + it('falls back to ranking alone when the agent predates the provenance resolver', async () => { + // The bridge talks to whatever agent the daemon composed; one without + // `resolveSyncPeerWithProvenance` must still rank the walk rather than + // throw — and must NOT infer an authority it cannot establish. + const { agent } = bridgeAgent({ + resolveSyncPeerWithProvenance: undefined, + resolvePreferredSyncPeerId: async () => 'peer-legacy', + }); + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-legacy']); + + expect(posted.result.preferredPeerId).toBe('peer-legacy'); + expect(posted.result.authoritativePeerId).toBeUndefined(); }); it('forwards the admission source into both detailed sync calls', async () => { diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 3fbedf3445..0ec1b4dec1 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -688,6 +688,12 @@ describe('catch-up plane proof predicates', () => { ['a timed-out phase', { timedOutPhases: 1 }], ['a denial', { deniedPhases: 1 }], ['a local admission deferral', { deferredBackpressure: 1 }], + // An integrity rejection is stronger than a failure: it is a peer that + // SERVED CONTENT for this graph which then failed verification, so it is + // positive evidence the graph is not empty. `classifyDurableProgress` + // already treats both as blocking failures per peer. + ['data rejected for missing metadata', { dataRejectedMissingMeta: 1 }], + ['a rejected Knowledge Collection', { rejectedKcs: 1 }], ])('voids the empty proof when the round contains %s', (_label, overrides) => { const diagnostics = { ...cleanEmptyRound, ...overrides }; expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) @@ -799,6 +805,20 @@ describe('catch-up plane proof predicates', () => { )).toBe(false); }); + it.each([ + ['data rejected for missing metadata', { dataRejectedMissingMeta: 1 }], + ['a rejected Knowledge Collection', { rejectedKcs: 1 }], + ])('is voided by %s elsewhere in the round, ahead of the curator\'s word', (_label, overrides) => { + // Content that failed verification still proves content EXISTS, which + // outranks the curator saying the graph is empty — unlike a plain + // transport or phase failure, which the curator's answer does outrank. + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, authorityEmptyPeers: 1 }, + { ...hostedEmptyDiagnostics, ...overrides }, + { isPrivate: false }, + )).toBe(false); + }); + it('never proves a private plane', () => { expect(catchupPlaneProvenByUnanimousEmpty( { ...noEvidence, authorityEmptyPeers: 1 }, From 19024a1f95b3b6a6caf519d76762e01b64722607 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:04:29 +0200 Subject: [PATCH 19/44] docs(changelog): state that rejected content voids the empty verdict Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 13392f15e1..2245af2f8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Fixed -- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. - **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`. - **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. From af3c38f0cb4e4c54e6b4cf9ec4454993785046e8 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:22:53 +0200 Subject: [PATCH 20/44] test(sync): pin every kill-switch spelling and the removed public constant MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 10. `DKG_CATCHUP_STOP_ON_PROOF` is documented with four disabled spellings but only `0` was ever driven, and `CATCHUP_STOP_ON_PROOF` resolves once at module load, so the other three could be dropped with every test still green — an operator who set `false` would silently keep running the fan-out they turned off. Extracted `resolveCatchupStopOnProof` so the operator contract is testable in place, same shape as the backpressure-budget parser. All four spellings, plus trimming and case-folding, plus default-ON for anything unrecognised so a typo cannot restore the pre-#2006 behaviour. Mutation: dropping `'false'` kills two, dropping the normalization kills five. The type contract covered the removed OPTION but not the removed EXPORT, so re-exporting `CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` from the package root would have regressed silently against a CHANGELOG that says stale callers must fail. Pinned with a `@ts-expect-error` root import, alongside a positive import of the replacement so the file cannot pass by the surface having decayed. Mutation: re-adding the export to the built `index.d.ts` fails the build. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../agent/src/sync/catchup-concurrency.ts | 24 +++++++++++--- .../agent/test/catchup-concurrency.test.ts | 31 +++++++++++++++++++ .../test/catchup-retry-contract.typecheck.ts | 12 ++++++- 3 files changed, 62 insertions(+), 5 deletions(-) diff --git a/packages/agent/src/sync/catchup-concurrency.ts b/packages/agent/src/sync/catchup-concurrency.ts index b86b6ed738..699ca00118 100644 --- a/packages/agent/src/sync/catchup-concurrency.ts +++ b/packages/agent/src/sync/catchup-concurrency.ts @@ -4,6 +4,24 @@ export const CATCHUP_MAX_CONCURRENT_PEER_SYNCS: number = (() => { return Number.isInteger(raw) && raw > 0 ? raw : 4; })(); +/** The spellings `DKG_CATCHUP_STOP_ON_PROOF` accepts as "off"; documented verbatim. */ +const CATCHUP_STOP_ON_PROOF_OFF_VALUES = ['0', 'false', 'no', 'off'] as const; + +/** + * Parse the progressive-walk kill-switch. + * + * Exported as a pure function because the constant below resolves once at + * module load, which makes the operator contract untestable in place — and the + * contract is four documented spellings plus trimming and case-folding, any one + * of which could be dropped without a single test noticing. Default is ON: + * anything unrecognised (including unset) leaves the walk enabled, so a typo + * cannot silently restore the pre-#2006 fan-out. + */ +export function resolveCatchupStopOnProof(raw: string | undefined): boolean { + const normalized = raw?.trim().toLowerCase(); + return !CATCHUP_STOP_ON_PROOF_OFF_VALUES.some((value) => value === normalized); +} + /** * Operator kill-switch for the progressive catch-up walk (issue #2006). * @@ -13,10 +31,8 @@ export const CATCHUP_MAX_CONCURRENT_PEER_SYNCS: number = (() => { * its own manifest, so stopping early can land one peer's snapshot instead of * the union of every peer's. */ -export const CATCHUP_STOP_ON_PROOF: boolean = (() => { - const raw = process.env.DKG_CATCHUP_STOP_ON_PROOF?.trim().toLowerCase(); - return !(raw === '0' || raw === 'false' || raw === 'no' || raw === 'off'); -})(); +export const CATCHUP_STOP_ON_PROOF: boolean = + resolveCatchupStopOnProof(process.env.DKG_CATCHUP_STOP_ON_PROOF); /** * Escalating wave sizes for the progressive peer walk: `startWidth`, ×2, ×2, … diff --git a/packages/agent/test/catchup-concurrency.test.ts b/packages/agent/test/catchup-concurrency.test.ts index 2cf583f429..e745ce05e0 100644 --- a/packages/agent/test/catchup-concurrency.test.ts +++ b/packages/agent/test/catchup-concurrency.test.ts @@ -1,7 +1,9 @@ import { describe, expect, it } from 'vitest'; import { CATCHUP_MAX_CONCURRENT_PEER_SYNCS, + CATCHUP_STOP_ON_PROOF, catchupWaveSizes, + resolveCatchupStopOnProof, } from '../src/sync/catchup-concurrency.js'; describe('catchupWaveSizes', () => { @@ -61,3 +63,32 @@ describe('catchupWaveSizes', () => { expect(CATCHUP_MAX_CONCURRENT_PEER_SYNCS).toBeGreaterThan(0); }); }); + +describe('resolveCatchupStopOnProof', () => { + // The kill-switch is operator-facing and documented with four disabled + // spellings. `CATCHUP_STOP_ON_PROOF` resolves once at module load, so without + // a pure parser only the spelling the suite happens to set is ever exercised + // — dropping `'false'` would leave an operator who set it silently running + // the very fan-out they turned off, with every test still green. + it.each(['0', 'false', 'no', 'off'])('treats %s as off', (value) => { + expect(resolveCatchupStopOnProof(value)).toBe(false); + }); + + it.each([' off ', 'OFF', 'False', 'No\t', ' 0'])('normalizes case and surrounding space in %j', (value) => { + expect(resolveCatchupStopOnProof(value)).toBe(false); + }); + + it.each([undefined, '', ' ', '1', 'true', 'yes', 'on', 'nope', 'offf', '0.0'])( + 'leaves the walk ON for %j', + (value) => { + // Default-on is the safe direction: an unrecognised value or a typo must + // not silently restore the pre-#2006 fan-out. + expect(resolveCatchupStopOnProof(value)).toBe(true); + }, + ); + + it('resolves the module constant through the same parser', () => { + expect(CATCHUP_STOP_ON_PROOF) + .toBe(resolveCatchupStopOnProof(process.env.DKG_CATCHUP_STOP_ON_PROOF)); + }); +}); diff --git a/packages/agent/test/catchup-retry-contract.typecheck.ts b/packages/agent/test/catchup-retry-contract.typecheck.ts index 1db4e2cfec..5640b7758b 100644 --- a/packages/agent/test/catchup-retry-contract.typecheck.ts +++ b/packages/agent/test/catchup-retry-contract.typecheck.ts @@ -1,9 +1,15 @@ import { + CATCHUP_BACKPRESSURE_MAX_WAIT_MS, runCatchupPlaneWithPolicy, type CatchupPlanePolicyClock, type CatchupPlanePolicyOptions, type CatchupPlaneResult, } from '@origintrail-official/dkg-agent'; +// @ts-expect-error CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS is REMOVED from the +// package root. It named the fixed [100, 250, 500] ladder, which no longer +// exists — re-exporting it would hand a consumer a schedule the node does not +// follow. A stale caller must fail to resolve it, not compile against a lie. +import { CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS } from '@origintrail-official/dkg-agent'; // `retryDelaysMs` configured the fixed `[100, 250, 500]` ladder that issue #2006 // replaced with a wall-clock budget. It is retained as `never` rather than @@ -50,8 +56,10 @@ const planes: CatchupPlanePolicyOptions retryDelaysMs: [10], }; -// The replacement is `retry.maxWaitMs`, and it must stay assignable. +// The replacements must stay importable and assignable, so this file cannot +// pass merely because the whole surface decayed. const supported: CatchupPlanePolicyClock = { retry: { maxWaitMs: 5_000 } }; +const replacementBudget: number = CATCHUP_BACKPRESSURE_MAX_WAIT_MS; export declare const pinned: [ typeof clock, @@ -59,5 +67,7 @@ export declare const pinned: [ typeof supported, typeof stale, typeof ladderIsUninhabited, + typeof replacementBudget, + typeof CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, typeof runCatchupPlaneWithPolicy, ]; From 9ac64b5db7a66838ddc8ec82536501b06cb85236 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:31:00 +0200 Subject: [PATCH 21/44] fix(sync): teach the pre-readiness gates about the authority-empty proof MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 11. Real gap, and the same bug class as #1921's "gate ALL verification consumers". `cleanCompletionHasResponse` gates the denial and no-response branches that run BEFORE `catchupPlaneReady` is ever consulted. Adding `authorityEmptyPeers` as a readiness proof without adding it there made the new evidence silently unreachable in exactly the shape it exists for: a public graph whose curator answers metadata-only, where any other peer's shared-memory phase gets refused, returned `denied` — discarding a durable plane the classifier would have proven ready. The predicate now lists every carrier of clean-completion evidence and says why it must, and takes the shared `CatchupPlaneCompletionEvidence` type rather than a structural duplicate, so a future carrier cannot be added to one and not the other without a type error. Regression test drives the reviewer's exact scenario. Mutation: removing the new clause from the gate kills it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/cli/src/context-graph-readiness.ts | 20 +++++++++++----- .../context-graph-catchup-readiness.test.ts | 23 +++++++++++++++++++ 2 files changed, 37 insertions(+), 6 deletions(-) diff --git a/packages/cli/src/context-graph-readiness.ts b/packages/cli/src/context-graph-readiness.ts index 874f0a2e3a..2c4362b42f 100644 --- a/packages/cli/src/context-graph-readiness.ts +++ b/packages/cli/src/context-graph-readiness.ts @@ -8,6 +8,7 @@ import { catchupPlaneCompletedWithoutFailure, catchupPlaneReady, type CatchupJobResult, + type CatchupPlaneCompletionEvidence, } from './catchup-runner.js'; export { catchupPlaneCompletedWithoutFailure } from './catchup-runner.js'; @@ -137,16 +138,23 @@ function catchupServedUsableData(result: CatchupJobResult): boolean { return result.dataSynced > 0 || result.sharedMemorySynced > 0; } +/** + * Did ANY peer complete this plane cleanly, whatever it carried? + * + * Every carrier of clean-completion evidence must be listed here, not just the + * ones that prove readiness: this predicate gates the denial and no-response + * branches that run BEFORE `catchupPlaneReady` is ever consulted, so a form of + * evidence missing from it is silently unreachable. The curator's hosted-empty + * round is the newest carrier and is exactly that shape — no data, no wire-empty + * response, and still a clean answer from the one peer that speaks for the graph. + */ function cleanCompletionHasResponse( - completion: { - verifiedDataPeers: number; - verifiedPrivateOnlyPeers?: number; - emptyPeers: number; - } | undefined, + completion: CatchupPlaneCompletionEvidence | undefined, ): boolean { return (completion?.verifiedDataPeers ?? 0) > 0 || (completion?.verifiedPrivateOnlyPeers ?? 0) > 0 || - (completion?.emptyPeers ?? 0) > 0; + (completion?.emptyPeers ?? 0) > 0 || + (completion?.authorityEmptyPeers ?? 0) > 0; } function catchupHasRequestedCleanPeerResponse( diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 6ac48c89f4..0fc8fbb49f 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -378,6 +378,29 @@ describe('context graph catch-up readiness classification', () => { }); }); + it('is not discarded by the denial gate before readiness is evaluated', () => { + // `cleanCompletionHasResponse` gates the denial and no-response branches + // that run BEFORE `catchupPlaneReady` is consulted. A new evidence carrier + // missing from that gate is silently unreachable: the durable plane would + // be provably ready and the job would still return `denied`, because a + // shared-memory phase from some other peer was refused. + const result = curatorHostedEmptyResult(); + result.denied = true; + result.deniedPeers = 1; + result.diagnostics!.sharedMemory.deniedPhases = 1; + + expect(classifyContextGraphCatchupReadiness({ + result, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', + readinessPatch: { durableVerified: true }, + }); + }); + it('never settles a PRIVATE plane on a curator hosted-empty round', () => { // Private planes stay proof-by-content only: an authorized-but-filtered // response is indistinguishable from an empty one on this side of the wire. From 753f0dcb985d47c6fdf538d92011e517b119f031 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:39:35 +0200 Subject: [PATCH 22/44] fix(sync): use the agent's own provenance contract at the worker bridge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 12. My own inconsistency: `9ac64b5db` introduced `authoritativeSyncPeerId` as "the single definition of which peer may end the walk", then the bridge did not use it — it re-shaped the resolution into `{ peerId?: string; provenance?: string }` and restated `provenance === 'metadata'` inline. A renamed or added provenance value would have kept compiling there and silently downgraded every curator to non-authoritative, at the one boundary where the distinction decides whether one peer can stop the entire walk. `SyncPeerResolution` and `authoritativeSyncPeerId` are now published from the agent root and consumed as-is. That is a deliberate exception to the surface shrinking asked for elsewhere: these are a genuine cross-package contract — the walk lives in the CLI worker while the resolver lives in the agent — not the retry-policy test seams that were correctly removed. The legacy fallback for an agent without `resolveSyncPeerWithProvenance` now states `provenance: 'bootstrap-hint'` explicitly rather than leaving it absent: an agent that cannot establish authority must not be assumed to have it. Mutation: bypassing the helper at the bridge kills two tests. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/index.ts | 10 ++++++++++ packages/cli/src/catchup-runner.ts | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index a9c2fb25cb..f9724c8f74 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -336,6 +336,16 @@ export { type CatchupPlanePolicyResult, type CatchupPlaneResult, } from './sync/catchup-policy.js'; +// Which peer may let one answer stand for a WHOLE Context Graph is the load- +// bearing distinction of the foreground catch-up walk (#2006), and the walk +// lives in the CLI's worker. Publishing the model — rather than letting the +// bridge re-shape it into a bare string — is what keeps the two sides from +// drifting: adding or renaming a provenance value must break the consumer, not +// silently downgrade it to "not authoritative". +export { + authoritativeSyncPeerId, + type SyncPeerResolution, +} from './dkg-agent-cg-resolve.js'; export { classifyDurableProgress, createFailedPeerDurableSyncResult, diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index f4f3445c3e..5896c2b4f0 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -2,12 +2,14 @@ import { Worker } from 'node:worker_threads'; import { existsSync } from 'node:fs'; import { fileURLToPath } from 'node:url'; import { + authoritativeSyncPeerId, classifyDurableProgress, normalizeDurableSyncResult, type DKGAgent, type DurableProgressSummary, type DurableSyncDiagnostics, type DurableSyncResult, + type SyncPeerResolution, } from '@origintrail-official/dkg-agent'; import { PROTOCOL_SYNC } from '@origintrail-official/dkg-core'; @@ -877,14 +879,20 @@ class WorkerCatchupRunner implements CatchupRunner { // the registry fallback twice for a wallet-address curator) per // catch-up, and the resolver evicts the bootstrap hint once metadata // confirms a curator, so the second call is not the same call. - const resolution: { peerId?: string; provenance?: string } = + const resolution: SyncPeerResolution = typeof agent.resolveSyncPeerWithProvenance === 'function' ? await agent.resolveSyncPeerWithProvenance(contextGraphId) - : { peerId: await agent.resolvePreferredSyncPeerId(contextGraphId) }; - const preferredPeerId: string | undefined = resolution.peerId; - const authoritativePeerId = resolution.provenance === 'metadata' - ? resolution.peerId - : undefined; + : { + peerId: await agent.resolvePreferredSyncPeerId(contextGraphId), + // An agent without the provenance resolver cannot establish + // authority, and must not be assumed to have it. + provenance: 'bootstrap-hint', + }; + const preferredPeerId = resolution.peerId; + // The agent's own definition of "may end the walk", not a restatement + // of it: a renamed or added provenance value has to break here rather + // than silently downgrade every curator to non-authoritative. + const authoritativePeerId = authoritativeSyncPeerId(resolution); if (preferredPeerId) { await agent.ensurePeerConnected(preferredPeerId); } From 1599470e4afc34d043c27af8dc8fd639a91dfc8c Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 13:49:30 +0200 Subject: [PATCH 23/44] test(sync): make the offline-curator test exercise a real offline authority MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 13. The fixture omitted `authoritativePeerId`, so the walk took the no-curator branch — already covered by the sibling test — and NEITHER half of `authorityFirst` was pinned. Deleting `syncCapable[0] === authoritativePeerId` failed nothing. The scenario the test names, a metadata-resolved curator the protocol probe then filters out, was never run. The fixture now sets it, and asserts the offline authority is never contacted while every reachable peer is. Dropping the comparison now narrows the opening wave to one and the peak-concurrency assertion fails. The `!== undefined` half survives mutation because it is unobservable: it differs only when there are no sync-capable peers, and `catchupWaveSizes(0, …)` is `[]`, so no waves run. Rather than delete a guard that keeps the degenerate `undefined === undefined` read from becoming reachable in a later refactor, it is now labelled defensive in place, pointing at the test that pins the invariant making it unobservable — so nobody reads it as a clause a test forgot to cover. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../cli/src/catchup-runner-worker-impl.ts | 8 +++++++ .../test/catchup-runner-worker-impl.test.ts | 24 ++++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index a563843e1d..988a6aaf86 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -407,6 +407,14 @@ async function runCatchup(request: CatchupRunRequest): Promise // add a round-trip to the front of every round, with no early stop to earn it // back. In that case the walk opens at the full concurrency cap — the previous // first-round latency. + // The `!== undefined` half is DEFENSIVE, not behavioural, and is called out + // as such so it does not read as a load-bearing clause a test should pin: + // with no resolvable curator and no sync-capable peers both sides are + // `undefined` and would compare equal, but `catchupWaveSizes(0, …)` is `[]` + // (pinned in `catchup-concurrency.test.ts`), so a zero-peer walk runs no + // waves and the opening width is unobservable. The comparison is what + // actually decides: narrow the opening wave only when the authority IS the + // peer that wave would contact. const authorityFirst = prepared.authoritativePeerId !== undefined && syncCapable[0] === prepared.authoritativePeerId; const waveSizes = CATCHUP_STOP_ON_PROOF diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index a351df3ff8..34452f6231 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -785,19 +785,34 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }); it('spends the single-peer opening wave only on a sync-capable curator', async () => { - // The curator is ranked first but is NOT sync-capable, so the walk has no - // authority to try alone and must not serialise an arbitrary peer instead. + // A REAL authority that is offline: metadata resolved a curator, so + // `authoritativePeerId` is set, but the protocol probe filters it out. The + // opening wave narrows to one peer only when the authority is the peer that + // wave would actually contact — otherwise the walk would serialise an + // arbitrary fallback peer for nothing. + // + // The fixture must set `authoritativePeerId`: without it the walk takes the + // no-curator branch (covered separately above) and neither half of the + // guard is exercised. const peerIds = ['peer-curator', 'peer-a', 'peer-b', 'peer-c', 'peer-d']; + const durableCalls: string[] = []; let inFlight = 0; let peak = 0; await runWorkerCatchup({ contextGraphId: 'cg-curator-offline', includeSharedMemory: false }, async (method, args) => { switch (method) { case 'prepareCatchup': - return { preferredPeerId: 'peer-curator', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; case 'waitForSyncProtocol': return args[0] !== 'peer-curator'; case 'syncDurable': { + durableCalls.push(args[0] as string); inFlight += 1; peak = Math.max(peak, inFlight); await delay(4); @@ -812,6 +827,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }); expect(peak).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + // The offline authority is never contacted, and every reachable peer is. + expect(durableCalls).not.toContain('peer-curator'); + expect([...durableCalls].sort()).toEqual(['peer-a', 'peer-b', 'peer-c', 'peer-d']); }); it('narrows fallback peers to the planes the curator already settled', async () => { From 5aacb7d18a4616bb8acd3c39f23a4805c0cd912e Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 14:04:46 +0200 Subject: [PATCH 24/44] fix(sync): a non-curator with only `_meta` can no longer prove a plane empty MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 14. Verified against a fresh build before and after: member-with-meta + stranger-empty => true (before) member-with-meta + stranger-empty => false (after) only strangers (no metadata) => true (unchanged) curator hosted-empty => true (unchanged) A peer that returns `_meta` and no data is the ambiguity this rule cannot resolve — the requester itself logs "peer may have empty or pruned data graph" for exactly that response. Combined with an unrelated peer's wire-empty answer it could still settle a public plane as `done` with zero Knowledge Assets, which is issue #2006's own headline symptom. `metaOnlyResponses > 0` now voids the whole-round verdict. This costs the legitimately-empty public graph nothing: when the curator IS present its own round settles the plane through the other proof mode, which is evaluated independently. And it is not a vacuous tightening — the all-strangers round still proves the plane, pinned by its own test, because a clause that can never be satisfied is worse than no clause. `catchupPlaneProvenByUnanimousEmpty` had grown two different proof modes behind one name. Split into `catchupPlaneProvenByAuthorityHostedEmpty` (the curator hosts it and it holds nothing) and the whole-round rule, with the shared "content exists, so it is not empty" checks factored into `emptyVerdictContradicted`. `catchupPlaneReady` composes the three in order of strength. Mutation: removing the `metaOnlyResponses` voider kills the new regression test; dropping the authority proof mode from `catchupPlaneReady` kills four. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- CHANGELOG.md | 2 +- packages/cli/src/catchup-runner.ts | 114 ++++++++++++++++------- packages/cli/test/catchup-runner.test.ts | 57 +++++++++++- 3 files changed, 134 insertions(+), 39 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2245af2f8c..daa4de3980 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Fixed -- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. So does a NON-curator answering `_meta` with no data — the requester itself logs "peer may have empty or pruned data graph" for that response, and without the curator present nothing can tell an empty graph from a member that has not synced it yet. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. - **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`. - **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 5896c2b4f0..b68b72495b 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -525,6 +525,8 @@ export interface CatchupPlaneRoundDiagnostics { fetchedMetaTriples?: number; fetchedDataTriples?: number; emptyResponses?: number; + /** Peers that returned `_meta` and no data; durable-only. */ + metaOnlyResponses?: number; failedPeers?: number; failedPhases?: number; timedOutPhases?: number; @@ -608,7 +610,59 @@ export function catchupPlaneProvenByData( } /** - * Whole-round proof that a public plane really is empty. + * Does any signal in this round rule out an empty verdict outright? + * + * Shared by BOTH empty-proof modes below, because these are not "a peer that + * failed" — they are evidence about the graph's contents that no peer's silence + * can outrank. + */ +function emptyVerdictContradicted( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, +): boolean { + // Verified content, obviously. + if (catchupPlaneProvenByData(completion)) return true; + // Data that arrived and was rejected. A peer SERVED CONTENT for this graph + // which then failed verification, so content exists even though we could not + // keep it. `classifyDurableProgress` already treats these as blocking + // failures per peer; this is the same rule applied to the round. + if ((diagnostics?.dataRejectedMissingMeta ?? 0) > 0 + || (diagnostics?.rejectedKcs ?? 0) > 0) return true; + // Data fetched anywhere in the round, whoever fetched it. + return (diagnostics?.fetchedDataTriples ?? 0) > 0; +} + +/** + * Proof mode 1 — the CURATOR hosts the graph and it holds nothing. + * + * A registered public graph that really is empty still carries definition + * triples in its own `/_meta`, so the peer hosting it answers + * metadata-only, never wire-empty, and could never satisfy the whole-round rule + * below. Its curator saying so is the only evidence such a graph can produce. + * + * Scoped to the metadata-resolved curator and nothing else. Any OTHER peer's + * metadata-only round is the commonest state on the network — a member that has + * `_meta` but has not synced the data yet — and accepting it would resettle + * issue #2006's exact failure as `done` with zero Knowledge Assets. + * + * Another peer merely failing part-way cannot contradict the curator; another + * peer producing CONTENT can, and that is what {@link emptyVerdictContradicted} + * checks — it means the curator's view is behind the network's. + */ +export function catchupPlaneProvenByAuthorityHostedEmpty( + completion: CatchupPlaneCompletionEvidence | undefined, + diagnostics: CatchupPlaneRoundDiagnostics | undefined, + options: { isPrivate: boolean }, +): boolean { + // Private planes stay proof-by-content only: an authorized-but-filtered + // response is indistinguishable from an empty one on this side of the wire. + if (options.isPrivate) return false; + if ((completion?.authorityEmptyPeers ?? 0) === 0) return false; + return !emptyVerdictContradicted(completion, diagnostics); +} + +/** + * Proof mode 2 — a whole round in which nobody had anything. * * A peer that has never heard of a Context Graph and a peer that hosts an empty * one are byte-identical on the wire: an unknown CG has no access policy, so the @@ -624,22 +678,20 @@ export function catchupPlaneProvenByData( * settled issue #2006's run as `done` with 1 KA out of 40, and either clause * kills it on its own. * - * A registered public graph that really is empty is therefore proven by its - * CURATOR instead — `authorityEmptyPeers`. Such a graph still carries - * definition triples in its own `/_meta`, so the peer hosting it answers - * metadata-only rather than wire-empty and could never satisfy the round rule - * above. Only the metadata-resolved curator counts: any OTHER peer's - * metadata-only round is the commonest state on the network — a member that - * has `_meta` but has not synced the data yet — and accepting it would resettle - * issue #2006's exact failure as `done` with zero Knowledge Assets. + * `metaOnlyResponses` also kills it. A non-curator that returned `_meta` and no + * data is the ambiguous case this rule cannot resolve — the requester itself + * logs "peer may have empty or pruned data graph" — and without the curator + * present there is nothing to resolve it against. When the curator IS present, + * proof mode 1 has already settled the plane, so voiding here costs the + * legitimately-empty graph nothing. * * Two counters are deliberately NOT consulted: * - * - `fetchedMetaTriples`. Every registered Context Graph carries definition - * triples in its own `/_meta`, so any peer that hosts the graph at all - * returns metadata even when the graph holds zero Knowledge Assets. Treating - * metadata as content would make a legitimately empty public graph - * permanently unreadable rather than merely unproven. + * - `fetchedMetaTriples`. A raw triple count, not a per-peer verdict: a delta + * sync legitimately carries the whole metadata phase with nothing newer than + * the watermark, and the requester deliberately does NOT flag that as + * metadata-only. Voiding on the raw count would make a legitimately empty + * public graph permanently unreadable rather than merely unproven. * - `failedPeers`. That is a transport failure to a peer we never heard from — * on a live testnet a majority of connected peers can be unreachable — and an * unreachable stranger is evidence of nothing. A peer that DID engage and @@ -658,26 +710,13 @@ export function catchupPlaneProvenByUnanimousEmpty( // Empty or metadata-only responses have never been able to prove that a // private graph is fully synchronized; that stays unchanged. if (options.isPrivate) return false; - if (catchupPlaneProvenByData(completion)) return false; - // An integrity rejection is not "a peer that failed" — it is a peer that - // SERVED CONTENT for this graph which then failed verification. That is - // positive evidence the graph is not empty, so it voids an empty verdict - // outright, ahead of even the curator's own word. `classifyDurableProgress` - // already treats these as blocking failures per peer; this is the same rule - // applied to the round. - if ((diagnostics?.dataRejectedMissingMeta ?? 0) > 0 - || (diagnostics?.rejectedKcs ?? 0) > 0) return false; - // The curator hosting the graph and carrying no data settles it on its own — - // it is the reference for the whole graph, so another peer merely failing - // part-way cannot contradict it. Another peer DELIVERING data can, which is - // the guard kept here: that means the curator's view is behind the network's. - if ((completion?.authorityEmptyPeers ?? 0) > 0) { - return (diagnostics?.fetchedDataTriples ?? 0) === 0; - } + if (emptyVerdictContradicted(completion, diagnostics)) return false; + // A non-curator that has `_meta` and no data cannot tell "the graph is empty" + // from "I have not synced it yet". See the note above. + if ((diagnostics?.metaOnlyResponses ?? 0) > 0) return false; const cleanEmptyObserved = (completion?.emptyPeers ?? 0) > 0 || (diagnostics?.emptyResponses ?? 0) > 0; if (!cleanEmptyObserved) return false; - if ((diagnostics?.fetchedDataTriples ?? 0) > 0) return false; return (diagnostics?.failedPhases ?? 0) === 0 && (diagnostics?.timedOutPhases ?? 0) === 0 && (diagnostics?.deniedPhases ?? 0) === 0 @@ -685,10 +724,14 @@ export function catchupPlaneProvenByUnanimousEmpty( } /** - * Canonical readiness proof for one catch-up plane. The peer walk stops early - * only on {@link catchupPlaneProvenByData}, so whenever this function falls - * through to the unanimous-empty branch the full peer set really was walked and - * the "nobody saw anything" denominator is meaningful. + * Canonical readiness proof for one catch-up plane: verified content, the + * curator's hosted-empty word, or a whole round in which nobody had anything — + * in that order of strength. + * + * The peer walk stops early only on {@link catchupPlaneProvenByData} or the + * curator's own round, so whenever this falls through to the unanimous-empty + * branch the full peer set really was walked and the "nobody saw anything" + * denominator is meaningful. */ export function catchupPlaneReady( completion: CatchupPlaneCompletionEvidence | undefined, @@ -696,6 +739,7 @@ export function catchupPlaneReady( options: { isPrivate: boolean }, ): boolean { return catchupPlaneProvenByData(completion) + || catchupPlaneProvenByAuthorityHostedEmpty(completion, diagnostics, options) || catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options); } diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 0ec1b4dec1..cee7b0f156 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -4,6 +4,7 @@ import { catchupPeerResponded, catchupPeerSucceeded, catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByAuthorityHostedEmpty, catchupPlaneProvenByData, catchupPlaneProvenByUnanimousEmpty, catchupPlaneReady, @@ -787,7 +788,7 @@ describe('catch-up plane proof predicates', () => { it('proves the public plane with no wire-empty response anywhere in the round', () => { const completion = { ...noEvidence, authorityEmptyPeers: 1 }; - expect(catchupPlaneProvenByUnanimousEmpty( + expect(catchupPlaneProvenByAuthorityHostedEmpty( completion, hostedEmptyDiagnostics, { isPrivate: false }, @@ -798,7 +799,7 @@ describe('catch-up plane proof predicates', () => { }); it('is voided when another peer delivered data the curator did not have', () => { - expect(catchupPlaneProvenByUnanimousEmpty( + expect(catchupPlaneProvenByAuthorityHostedEmpty( { ...noEvidence, authorityEmptyPeers: 1 }, { ...hostedEmptyDiagnostics, fetchedDataTriples: 122_705 }, { isPrivate: false }, @@ -812,7 +813,7 @@ describe('catch-up plane proof predicates', () => { // Content that failed verification still proves content EXISTS, which // outranks the curator saying the graph is empty — unlike a plain // transport or phase failure, which the curator's answer does outrank. - expect(catchupPlaneProvenByUnanimousEmpty( + expect(catchupPlaneProvenByAuthorityHostedEmpty( { ...noEvidence, authorityEmptyPeers: 1 }, { ...hostedEmptyDiagnostics, ...overrides }, { isPrivate: false }, @@ -828,6 +829,56 @@ describe('catch-up plane proof predicates', () => { }); }); + describe('a non-curator that has `_meta` but no data', () => { + // The requester itself logs "peer may have empty or pruned data graph" for + // this response, which names the ambiguity exactly: the graph is empty, OR + // this member has not synced it yet. Without the curator present there is + // nothing to resolve it against, and combining it with an unrelated peer's + // empty answer would settle a 40-KA graph as `done` with zero. + const memberWithMetaOnly = { + ...cleanEmptyRound, + emptyResponses: 1, + metaOnlyResponses: 1, + fetchedMetaTriples: 9, + }; + + it('cannot be combined with a stranger\'s empty answer to prove the plane', () => { + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(false); + expect(catchupPlaneReady( + { ...noEvidence, emptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(false); + }); + + it('costs the legitimately empty graph nothing once its CURATOR answers', () => { + // The positive half. Voiding on `metaOnlyResponses` would be a bad trade + // if it also blocked the real empty-public-graph case — it does not, + // because the curator's own round settles that through the other proof + // mode, which is evaluated independently. + expect(catchupPlaneReady( + { ...noEvidence, emptyPeers: 1, authorityEmptyPeers: 1 }, + memberWithMetaOnly, + { isPrivate: false }, + )).toBe(true); + }); + + it('leaves the all-strangers round provable, so the rule is not vacuous', () => { + // A tightened clause that can never be satisfied is worse than no clause, + // because nothing reveals it. Pin that the unanimous rule still fires + // when every responder answered wire-empty and nobody returned metadata. + expect(catchupPlaneProvenByUnanimousEmpty( + { ...noEvidence, emptyPeers: 2 }, + { ...cleanEmptyRound, metaOnlyResponses: 0 }, + { isPrivate: false }, + )).toBe(true); + }); + }); + it('accepts either evidence carrier for the clean empty completion', () => { // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter // (`diagnostics.emptyResponses`) are separate carriers, and the legacy From 96e1c9bbd5cf65e732aae03d934b03c7a6e9b912 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 14:12:23 +0200 Subject: [PATCH 25/44] test(sync): pin every admission source at its production call site MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 15. The source labels were proven to normalize and display correctly, but not to be SUPPLIED. Deleting `source: 'vm-recovery'` from `syncExactKnowledgeAssetsFromPeer` left every test green — the existing assertion matched `exactAssetUals`, `stopOnBackoffWorthyFailure` and `priority` but not `source` — so the regression would have shown up only as `durable:unspecified` on an operator's dashboard. Swept the class rather than the instance. Of the seven declared `SYNC_ADMISSION_SOURCES`, three were unpinned at their call sites: - `vm-recovery` — added to the existing exact-recovery assertion. - `swm-recovery` — no coverage at all; new test drives the real `recoverContextGraphSwmFromPeer` prototype and asserts lane + admission. - `on-connect` — no coverage at all, and the highest-volume source in production: every reconnect sync flows through that default parameter. Each mutation-checked by deleting the label at its call site. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../durable-sync-lifecycle-binding.test.ts | 32 +++++++++++++++++++ .../agent/test/sync-on-connect-churn.test.ts | 25 +++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/packages/agent/test/durable-sync-lifecycle-binding.test.ts b/packages/agent/test/durable-sync-lifecycle-binding.test.ts index f07c53cfce..6a42369a4a 100644 --- a/packages/agent/test/durable-sync-lifecycle-binding.test.ts +++ b/packages/agent/test/durable-sync-lifecycle-binding.test.ts @@ -335,10 +335,42 @@ describe('durable sync lifecycle chain binding', () => { exactAssetUals: [exactUal], stopOnBackoffWorthyFailure: true, priority: 1_000, + // The admission SOURCE is what makes this show up as `durable:vm-recovery` + // rather than `durable:unspecified` on the sync-global scheduler, which is + // the whole point of the label. Without this line, deleting it from the + // call site keeps every test green and only the Grafana attribution rots. + source: 'vm-recovery', }); expect(runLegacyDurableSync.mock.calls[0]?.[6]).not.toHaveProperty('totalTimeoutMs'); }); + it('labels standalone SWM recovery admissions at the call site', async () => { + // The sibling of the VM-recovery assertion above, and the one that had NO + // coverage: a regression dropping this source would report SWM recovery + // pressure as `shared-memory:unspecified` on the sync-global scheduler. + // Asserted on the real prototype so it pins the production call site, not a + // re-statement of it. + const runContextGraphSyncWithBackpressure = vi.fn(async () => ({})); + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + runContextGraphSyncWithBackpressure, + }; + + await LifecycleSyncMethods.prototype.recoverContextGraphSwmFromPeer.call( + agentLike as any, + '12D3KooWSwmRecoveryPeer', + 'private-cg', + ); + + expect(runContextGraphSyncWithBackpressure).toHaveBeenCalledTimes(1); + const [, contextGraphId, lane, , , admission] = + runContextGraphSyncWithBackpressure.mock.calls[0] as unknown as unknown[]; + expect(contextGraphId).toBe('private-cg'); + expect(lane).toBe('swm_recovery'); + expect(admission).toEqual({ source: 'swm-recovery' }); + }); + it('honors an explicit exact-asset timeout while internal VM recovery keeps 600 seconds', async () => { vi.spyOn(Date, 'now').mockReturnValue(1_800_000_000_000); const exactUal = 'did:dkg:base:84532/0x1111111111111111111111111111111111111111/1'; diff --git a/packages/agent/test/sync-on-connect-churn.test.ts b/packages/agent/test/sync-on-connect-churn.test.ts index fc786c95df..ae3fbe1d31 100644 --- a/packages/agent/test/sync-on-connect-churn.test.ts +++ b/packages/agent/test/sync-on-connect-churn.test.ts @@ -118,6 +118,31 @@ describe('sync-on-connect churn gates', () => { expect(calls).toEqual([PEER_A]); }); + it('labels the connect-driven admission on-connect, not unspecified', async () => { + // The complement of the reconciler assertion below, and the source with the + // highest production volume: every reconnect sync flows through this + // default. Nothing pinned it, so changing the default would silently + // relabel most sync-global pressure on the operator dashboards. + const agent = await createUnstartedAgent('SyncOnConnectSourceLabel'); + (agent as any).started = true; + const sources: unknown[] = []; + (agent as any).trySyncFromPeer = async ( + _peer: string, + _onAccounting: unknown, + source: unknown, + ) => { + sources.push(source); + return undefined; + }; + + await (agent as any).attemptSyncFromPeerWithReconcilerAccounting(PEER_A, { + connected: true, + hasSyncProtocol: true, + }); + + expect(sources).toEqual(['on-connect']); + }); + it('reconciler still retries stale connected peers', async () => { const agent = await createUnstartedAgent('SyncReconcilerStillRetries'); (agent as any).started = true; From 330a9d62ebd4df23ace71d1ef57688a19e920973 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 14:31:12 +0200 Subject: [PATCH 26/44] fix(sync): contain the untrusted admission source at the worker boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 16. `source` was typed `string` across the whole in-process sync API because ONE producer — the catch-up Worker RPC — delivers whatever crossed a structured-clone boundary. That let the untrusted edge's looseness leak into every ordinary caller: `syncFromPeerDetailed({ source: 'durable:urn:cg:...' })` type-checked, even though admission sources are a closed diagnostic set and an identifier-bearing label would become a metric dimension. The clamp now happens where the untrusted value actually enters — the CLI bridge decodes the RPC argument as `unknown` and runs `normalizeSyncAdmissionSource` before calling in — and the three agent options carry `SyncAdmissionSource`. `runContextGraphSyncWithBackpressure` still re-clamps. That is deliberate rather than redundant: it is the single choke point every admission passes through, and a clamp that cannot be bypassed is worth more than one that merely type-checks. New test drives an identifier-bearing string and a non-string through the real bridge and asserts both arrive as `unspecified`. Mutation: bypassing the clamp kills it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-lifecycle.ts | 31 ++++++++++--------- packages/cli/src/catchup-runner.ts | 23 +++++++++++--- .../catchup-runner-worker-lifecycle.test.ts | 15 +++++++++ 3 files changed, 50 insertions(+), 19 deletions(-) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 0247123adf..aa096a652a 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -951,14 +951,14 @@ export type DurableSyncOptions = { * node-wide scheduler diagnostics so queue pressure can be attributed to an * origin. * - * Typed `string`, not `SyncAdmissionSource`, on purpose: these options are - * reconstructed from a `postMessage` payload after crossing the catch-up - * Worker RPC boundary, where the compile-time union guarantees nothing. This - * is the untrusted edge; `normalizeSyncAdmissionSource` clamps it to the - * closed set once, in `acquire`, and every layer past that clamp is typed - * `SyncAdmissionSource`. + * The closed union, so an ordinary in-process caller cannot introduce an + * unbounded or identifier-bearing label. The catch-up Worker RPC is the one + * path where the compile-time union guarantees nothing — a `postMessage` + * payload is whatever crossed the wire — and that edge clamps with + * `normalizeSyncAdmissionSource` in the CLI bridge before calling in. + * The scheduler re-clamps anyway, as defence in depth. */ - source?: string; + source?: SyncAdmissionSource; }; type LegacyDurableContextGraphOptions = { @@ -1177,11 +1177,12 @@ export class LifecycleSyncMethods extends DKGAgentBase { priorityOverride?: number; operationSignal?: AbortSignal; /** - * Which trigger enqueued this admission. Accepted as a loose string - * because it can arrive from the catch-up Worker RPC; normalized to the - * closed set HERE so every layer past this boundary carries the union. + * Which trigger enqueued this admission. Typed as the closed union for + * ordinary callers; still normalized HERE, because this is the single + * choke point every admission passes through and a clamp that cannot be + * bypassed is worth more than one that merely type-checks. */ - source?: string; + source?: SyncAdmissionSource; } = {}, ): Promise { const { priorityOverride, operationSignal } = admission; @@ -5254,11 +5255,11 @@ export class LifecycleSyncMethods extends DKGAgentBase { /** Admission override for foreground catch-up. */ priority?: number; /** - * Bounded admission origin for node-wide scheduler diagnostics. `string` - * because it can arrive across the catch-up Worker RPC boundary; clamped - * to the closed `SyncAdmissionSource` set in `acquire`. + * Bounded admission origin for node-wide scheduler diagnostics. The + * closed union: the catch-up Worker RPC is the only untrusted producer + * and it clamps in the CLI bridge, while `acquire` re-clamps regardless. */ - source?: string; + source?: SyncAdmissionSource; }, ): Promise { const ctx = createOperationContext('sync'); diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index b68b72495b..4bc40ccd2e 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -5,6 +5,7 @@ import { authoritativeSyncPeerId, classifyDurableProgress, normalizeDurableSyncResult, + normalizeSyncAdmissionSource, type DKGAgent, type DurableProgressSummary, type DurableSyncDiagnostics, @@ -967,7 +968,7 @@ class WorkerCatchupRunner implements CatchupRunner { } case 'syncDurable': { const [peerId, contextGraphId, priority, source] = args as [ - string, string, number | undefined, string | undefined, + string, string, number | undefined, unknown, ]; return agent.syncFromPeerDetailed( peerId, @@ -975,17 +976,31 @@ class WorkerCatchupRunner implements CatchupRunner { undefined, undefined, undefined, - { ...(priority === undefined ? {} : { priority }), source }, + { + ...(priority === undefined ? {} : { priority }), + // This RPC argument crossed a structured-clone boundary, so its + // compile-time type guaranteed nothing. Clamp it to the closed + // diagnostic set HERE, at the untrusted edge, so every in-process + // caller past it is typed `SyncAdmissionSource`. + source: normalizeSyncAdmissionSource( + typeof source === 'string' ? source : undefined, + ), + }, ); } case 'syncSharedMemory': { const [peerId, contextGraphId, priority, source] = args as [ - string, string, number | undefined, string | undefined, + string, string, number | undefined, unknown, ]; return agent.syncSharedMemoryFromPeerDetailed( peerId, [contextGraphId], - { ...(priority === undefined ? {} : { priority }), source }, + { + ...(priority === undefined ? {} : { priority }), + source: normalizeSyncAdmissionSource( + typeof source === 'string' ? source : undefined, + ), + }, ); } case 'finalizeCatchup': { diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index 2463616efe..c82c5fa478 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -213,6 +213,21 @@ describe('WorkerCatchupRunner agent bridge', () => { expect(posted.result.authoritativePeerId).toBeUndefined(); }); + it('clamps an unbounded RPC source at the untrusted edge', async () => { + // A worker RPC argument crossed a structured-clone boundary, so its + // compile-time type guaranteed nothing. If an identifier-bearing value + // reached the scheduler it would become a metric and log DIMENSION, + // re-opening the correlation-identifier leak that collapsing the operation + // label was added to close, and multiplying diagnostic cardinality. + const { agent, calls } = bridgeAgent(); + + await invokeThroughBridge(agent, 'syncDurable', ['peer-a', 'cg-x', 2000, 'durable:urn:cg:private:abc']); + await invokeThroughBridge(agent, 'syncSharedMemory', ['peer-a', 'cg-x', 2000, { not: 'a string' }]); + + expect(calls.durable[0]!.at(-1)).toMatchObject({ source: 'unspecified' }); + expect(calls.shared[0]!.at(-1)).toMatchObject({ source: 'unspecified' }); + }); + it('forwards the admission source into both detailed sync calls', async () => { const { agent, calls } = bridgeAgent(); From f58c8273c7a3affdc9fa7ae0675076c0b4161d43 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 14:45:17 +0200 Subject: [PATCH 27/44] test(sync): cover the bridge handoff that produces the authority-ranked list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 17. Same class as round 13 and round 15: the walk's whole load reduction depends on opening with the curator, but the worker only ever sees an ALREADY-ranked `peerIds`, and every worker test supplies that list itself. The handoff that produces it — resolve the peer, then rank the live connections against it — was covered nowhere. New bridge test drives real `prepareCatchup` with out-of-order, duplicated fake connections and a `selectCatchupPeers` spy. It asserts the resolved peer is what selection ranks against, that the worker receives ranked plain-string ids with the curator first, and that `authoritativePeerId` travels alongside. Mutation kills two, one of them unplanned: - dropping `preferredPeerId` from the `selectCatchupPeers` call - dropping the connection de-duplication, which nothing had covered either — a peer with two live connections was counted twice in `connectedPeers` Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../catchup-runner-worker-lifecycle.test.ts | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index c82c5fa478..eee243188d 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -228,6 +228,52 @@ describe('WorkerCatchupRunner agent bridge', () => { expect(calls.shared[0]!.at(-1)).toMatchObject({ source: 'unspecified' }); }); + it('hands the resolved peer into selection and returns an authority-ranked list', async () => { + // The whole load reduction depends on the walk opening with the curator, + // and the worker only ever sees an ALREADY-ranked `peerIds`. Every + // worker test supplies that list itself, so the handoff that produces it — + // resolve, then rank the live connections against the resolved peer — was + // covered nowhere. Dropping the second argument here would leave the worker + // suite green while production opened at the full cap. + const selectCalls: unknown[][] = []; + const { agent } = bridgeAgent({ + resolveSyncPeerWithProvenance: async () => ({ + peerId: 'peer-curator', + provenance: 'metadata', + }), + node: { + libp2p: { + getConnections: () => ['peer-b', 'peer-curator', 'peer-a', 'peer-b'].map( + (id) => ({ remotePeer: { toString: () => id } }), + ), + }, + }, + selectCatchupPeers: (...args: unknown[]) => { + selectCalls.push(args); + const peers = args[0] as Array<{ toString(): string }>; + const preferred = args[1] as string | undefined; + // Stand in for the real ranking: preferred first, rest in order. + return [...peers].sort((a, b) => Number(b.toString() === preferred) + - Number(a.toString() === preferred)); + }, + }); + + const posted = await invokeThroughBridge(agent, 'prepareCatchup', ['cg-rank']); + + expect(selectCalls).toHaveLength(1); + const [candidates, preferred, isPrivate] = selectCalls[0]!; + // Live connections are de-duplicated before ranking… + expect((candidates as Array<{ toString(): string }>).map((p) => p.toString())) + .toEqual(['peer-b', 'peer-curator', 'peer-a']); + // …the RESOLVED peer is what selection ranks against… + expect(preferred).toBe('peer-curator'); + expect(isPrivate).toBe(false); + // …and the worker receives the ranked ids as plain strings, curator first. + expect(posted.result.peerIds).toEqual(['peer-curator', 'peer-b', 'peer-a']); + expect(posted.result.authoritativePeerId).toBe('peer-curator'); + expect(posted.result.connectedPeers).toBe(3); + }); + it('forwards the admission source into both detailed sync calls', async () => { const { agent, calls } = bridgeAgent(); From 161332fdc673a00ec6bf8d17bbbf929c9cb66fd4 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 14:58:00 +0200 Subject: [PATCH 28/44] fix(sync): shared-memory metadata is not hosted-empty proof MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 18. The hosted-empty proof was plane-agnostic, but "metadata means the peer hosts the graph" is a DURABLE fact: `/_meta` carries the Context Graph's own definition triples, so serving it proves hosting. Shared-memory metadata is a different artifact, and shared memory is contributed by many members rather than owned by the curator — so "the curator has SWM structure but no SWM rows" does not mean the network has none. Left generic, a curator's clean SWM round with `insertedMetaTriples > 0` and no data set `authorityEmptyPeers = 1`, settled the shared plane, and stopped the walk before any member holding the SWM rows was contacted. On the shared plane only a genuine wire-empty response now counts; the durable plane is unchanged. Two tests, unit and end-to-end, and both mutations die: - removing the plane discriminator from the reducer - the worker call site forgetting to declare `plane: 'shared-memory'` Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../cli/src/catchup-runner-worker-impl.ts | 3 +- packages/cli/src/catchup-runner.ts | 23 +++++++-- .../test/catchup-runner-worker-impl.test.ts | 47 +++++++++++++++++++ packages/cli/test/catchup-runner.test.ts | 36 ++++++++++++++ 4 files changed, 104 insertions(+), 5 deletions(-) diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 988a6aaf86..065ee8b331 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -307,6 +307,7 @@ async function runCatchup(request: CatchupRunRequest): Promise const durableEvidence = catchupPeerPlaneEvidence(durable, { complete: durable.complete, fromAuthority, + plane: 'durable', }); addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); // The curator answering cleanly settles this plane whether it carried @@ -345,7 +346,7 @@ async function runCatchup(request: CatchupRunRequest): Promise // Shared memory carries no verified-private-only signal, so the shared // evidence only ever has data/empty set — the same reducer still applies. - const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority }); + const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority, plane: 'shared-memory' }); addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); // Same rule as durable: the curator settles the plane by answering // cleanly, with data or empty. Shared memory is frequently empty for a diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 4bc40ccd2e..ec1729e70b 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -554,7 +554,15 @@ export function catchupPeerPlaneEvidence( | (CatchupPhaseProgress & { emptyResponses?: number; fetchedDataTriples?: number }) | null | undefined, - options: { complete?: boolean; fromAuthority?: boolean } = {}, + options: { + complete?: boolean; + fromAuthority?: boolean; + /** + * Which plane this result came from. Required for authority evidence + * because "metadata proves the peer hosts the graph" is a DURABLE fact. + */ + plane?: 'durable' | 'shared-memory'; + } = {}, ): CatchupPlaneCompletionEvidence { const none = { verifiedDataPeers: 0, @@ -570,9 +578,16 @@ export function catchupPeerPlaneEvidence( // peer does not have it. const carriedNoData = (plane.insertedDataTriples ?? 0) === 0 && (plane.fetchedDataTriples ?? 0) === 0; - const answered = (plane.emptyResponses ?? 0) > 0 - || (plane.metaOnlyResponses ?? 0) > 0 - || (plane.insertedMetaTriples ?? 0) > 0; + // Metadata counts as hosting evidence on the DURABLE plane only. There, + // `/_meta` carries the Context Graph's own definition triples, so serving + // it proves the peer hosts the graph. Shared-memory metadata is a different + // artifact and carries no such guarantee — and shared memory is contributed by + // many members rather than owned by the curator, so "the curator has SWM + // structure but no SWM rows" does not mean the network has none. On that + // plane, only a genuine wire-empty response counts. + const hostsGraph = options.plane !== 'shared-memory' + && ((plane.metaOnlyResponses ?? 0) > 0 || (plane.insertedMetaTriples ?? 0) > 0); + const answered = (plane.emptyResponses ?? 0) > 0 || hostsGraph; return { verifiedDataPeers: (plane.insertedDataTriples ?? 0) > 0 ? 1 : 0, verifiedPrivateOnlyPeers: (plane.verifiedPrivateOnlyResponses ?? 0) > 0 ? 1 : 0, diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 34452f6231..503777508f 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -622,6 +622,53 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(1); }); + it('does not settle the SHARED-MEMORY plane on curator metadata alone', async () => { + // End-to-end counterpart of the plane-aware reducer: shared memory is + // contributed by many members rather than owned by the curator, so + // `insertedMetaTriples` there is not the hosting proof `/_meta` is on + // the durable plane. Settling on it would stop the walk before any member + // holding the SWM rows is contacted. Public graph, so privacy is not what + // is doing the work here. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-meta', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-0', + authoritativePeerId: 'peer-0', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + return { + ...sharedResult(), + insertedTriples: 5, + insertedMetaTriples: 5, + insertedDataTriples: 0, + fetchedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 0, + completedPhases: 2, + }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); + }); + it('settles a public plane when the CURATOR hosts the graph and has no data', async () => { // A registered public Context Graph with no Knowledge Assets yet. Its host // still serves the CG definition triples from `/_meta`, so it answers diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index cee7b0f156..ada021273e 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -769,6 +769,42 @@ describe('catch-up plane proof predicates', () => { emptyResponses: 0, }; + it('does not read SHARED-MEMORY metadata as hosting evidence', () => { + // `/_meta` definition triples are a DURABLE fact: serving them proves + // the peer hosts the Context Graph. Shared-memory metadata is a different + // artifact, and shared memory is contributed by many members rather than + // owned by the curator — so "the curator has SWM structure but no SWM + // rows" does not mean the network has none. Treating it as hosted-empty + // would settle the shared plane and stop the walk before any member that + // actually holds the SWM data is contacted. + const curatorSharedMetaOnly = { + insertedTriples: 5, + insertedMetaTriples: 5, + insertedDataTriples: 0, + fetchedDataTriples: 0, + emptyResponses: 0, + completedPhases: 2, + }; + + expect(catchupPeerPlaneEvidence(curatorSharedMetaOnly, { + fromAuthority: true, + plane: 'shared-memory', + })).toMatchObject({ authorityEmptyPeers: 0 }); + + // A genuine wire-empty shared-memory round from the curator still counts. + expect(catchupPeerPlaneEvidence( + { ...curatorSharedMetaOnly, insertedTriples: 0, insertedMetaTriples: 0, emptyResponses: 1 }, + { fromAuthority: true, plane: 'shared-memory' }, + )).toMatchObject({ authorityEmptyPeers: 1 }); + + // …and the identical shape on the DURABLE plane is hosting evidence. + expect(catchupPeerPlaneEvidence(curatorSharedMetaOnly, { + complete: true, + fromAuthority: true, + plane: 'durable', + })).toMatchObject({ authorityEmptyPeers: 1 }); + }); + it('counts the curator, and ONLY the curator, as hosted-empty evidence', () => { expect(catchupPeerPlaneEvidence(hostedEmptyRound, { complete: true, From 1ee5c1371a65441b05808e2512f5bcfb129e8d4a Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 15:23:40 +0200 Subject: [PATCH 29/44] fix(sync): the curator is not authoritative for a PUBLIC shared-memory plane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 19. `161332fdc` stated this principle and only half-applied it: it stopped shared-memory METADATA counting as hosting evidence, but a curator whose SWM round was wire-empty still produced `authorityEmptyPeers: 1`, settled the plane, and stopped the walk. The codebase is explicit that this is wrong for public graphs: planSharedMemorySyncContextGraphs — "a PRIVATE CG converges by REPLACE-recovering the current state from its CURATOR (the authoritative SWM replica) ... PUBLIC CGs keep the union path" and it is load-bearing, not just intent: `applyCuratorScope` narrows the SWM catch-up peer set to curator peers ONLY when the graph is private, a public CG has no authoritative member roster at all, and the sync responder serves the SWM plane purely from its own local store. So a curator answering zero SWM rows means "this node holds none", never "the network holds none". Host mode does not change it — hosts are arbitrary connected cores keeping a TTL- and byte-capped FIFO, and host catch-up is the fallback for when syncing from members returns nothing. Shared-memory rounds therefore produce no hosted-empty evidence at all now. The plane is still provable: by verified DATA from any peer, or as a whole-round verdict once every peer has answered — which is reachable again precisely because the walk no longer stops. The amplification fix is unaffected. It comes from per-plane narrowing, not from the break: once the curator settles durable, every fallback peer takes the `durable: null` branch, so the 147,246-triple / ~278 MB durable re-pull stays removed. What is given up is the early break on a subscribe whose graph has no shared memory — cheap wire-empty rounds, in exchange for not skipping a member that holds rows. Mutation: allowing the shared plane to produce authority-empty evidence again kills four tests, including a new end-to-end one where the curator is empty and peer-1 has the rows. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- CHANGELOG.md | 2 +- packages/cli/src/catchup-runner.ts | 35 +++++--- .../test/catchup-runner-worker-impl.test.ts | 79 +++++++++++++++---- packages/cli/test/catchup-runner.test.ts | 8 +- 4 files changed, 96 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index daa4de3980..a8517811d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Fixed -- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. So does a NON-curator answering `_meta` with no data — the requester itself logs "peer may have empty or pruned data graph" for that response, and without the curator present nothing can tell an empty graph from a member that has not synced it yet. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. +- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled — which is what removes the amplification, since the peers still contacted skip the plane already served. What the curator may settle differs by plane: it settles either plane by delivering verified DATA, but it settles a plane by being EMPTY only for durable data, which it owns. Shared memory is a per-agent-address layered union contributed by many members (`PUBLIC CGs keep the union path`), so a curator holding no shared-memory rows has said nothing about the members' layers; an empty shared-memory plane is still provable, but only as a whole-round verdict once every peer has answered. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. So does a NON-curator answering `_meta` with no data — the requester itself logs "peer may have empty or pruned data graph" for that response, and without the curator present nothing can tell an empty graph from a member that has not synced it yet. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network. - **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`. - **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status. diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index ec1729e70b..ac75be7f32 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -578,16 +578,31 @@ export function catchupPeerPlaneEvidence( // peer does not have it. const carriedNoData = (plane.insertedDataTriples ?? 0) === 0 && (plane.fetchedDataTriples ?? 0) === 0; - // Metadata counts as hosting evidence on the DURABLE plane only. There, - // `/_meta` carries the Context Graph's own definition triples, so serving - // it proves the peer hosts the graph. Shared-memory metadata is a different - // artifact and carries no such guarantee — and shared memory is contributed by - // many members rather than owned by the curator, so "the curator has SWM - // structure but no SWM rows" does not mean the network has none. On that - // plane, only a genuine wire-empty response counts. - const hostsGraph = options.plane !== 'shared-memory' - && ((plane.metaOnlyResponses ?? 0) > 0 || (plane.insertedMetaTriples ?? 0) > 0); - const answered = (plane.emptyResponses ?? 0) > 0 || hostsGraph; + // Whose emptiness counts, and on which plane. + // + // DURABLE: the Context Graph is the curator's. `/_meta` carries its own + // definition triples, so a curator serving them proves it hosts the graph, and + // a curator with no data means the graph has none. Both wire-empty and + // metadata-only rounds are hosted-empty evidence there. + // + // SHARED MEMORY: nobody's emptiness counts, not even the curator's. SWM is a + // per-agent-address layered union (`//`) contributed by many + // members, so a curator holding no SWM rows says nothing about the members' + // layers — it does not own them. Letting it settle the plane skipped peers that + // held valid rows and could report `sharedMemoryVerified` with + // `sharedMemorySynced: 0`. An empty SWM plane is still provable, but only as a + // WHOLE-ROUND verdict once every peer has answered, which is what + // `catchupPlaneProvenByUnanimousEmpty` is for. + // + // Verified DATA from the curator still settles either plane. That is the + // tradeoff this PR states openly — a peer's `complete` flag proves only its own + // manifest, and the background reconciler remains the convergence mechanism — + // and it is what keeps the amplification fixed for an SWM-heavy graph, which + // issue #2006 measured at 122,705 fetched triples on the shared plane alone. + const answered = options.plane !== 'shared-memory' + && ((plane.emptyResponses ?? 0) > 0 + || (plane.metaOnlyResponses ?? 0) > 0 + || (plane.insertedMetaTriples ?? 0) > 0); return { verifiedDataPeers: (plane.insertedDataTriples ?? 0) > 0 ? 1 : 0, verifiedPrivateOnlyPeers: (plane.verifiedPrivateOnlyResponses ?? 0) > 0 ? 1 : 0, diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 503777508f..614dac4318 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -363,16 +363,14 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.peersNotAttempted).toBe(peerIds.length - 1); }); - it('lets the curator settle a plane by answering cleanly empty', async () => { - // Shared memory is frequently empty for a graph that has durable data, and - // `includeSharedMemory` defaults to true on subscribe. If only inserted - // rows could settle a plane, the early stop would almost never fire in the - // shape this fix targets. + it('lets the curator settle the DURABLE plane by answering cleanly empty', async () => { + // The Context Graph is the curator's, so its "there is nothing here" is + // authoritative for the durable plane and one payload settles it. The + // shared-memory plane is deliberately NOT symmetric — see the next test. const peerIds = Array.from({ length: 10 }, (_, i) => `peer-${i}`); const durableCalls: string[] = []; - const sharedCalls: string[] = []; - const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-swm', includeSharedMemory: true }, async (method, args) => { + const result = await runWorkerCatchup({ contextGraphId: 'cg-empty-durable', includeSharedMemory: false }, async (method, args) => { switch (method) { case 'prepareCatchup': return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; @@ -380,11 +378,8 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) return true; case 'syncDurable': durableCalls.push(args[0] as string); - return durableResult(); - case 'syncSharedMemory': - sharedCalls.push(args[0] as string); return { - ...sharedResult(), + ...durableResult(), insertedTriples: 0, fetchedDataTriples: 0, insertedDataTriples: 0, @@ -400,9 +395,62 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) }); expect(durableCalls).toEqual(['peer-0']); - expect(sharedCalls).toEqual(['peer-0']); expect(result.peersNotAttempted).toBe(peerIds.length - 1); - expect(result.cleanPlaneCompletions?.sharedMemory.emptyPeers).toBe(1); + expect(result.cleanPlaneCompletions?.durable.authorityEmptyPeers).toBe(1); + }); + + it('does NOT let the curator settle the shared-memory plane by answering empty', async () => { + // Shared memory is a per-agent-address layered union + // (`//`) contributed by many members, so a curator that + // holds no SWM rows has not said anything about the members' layers — it + // does not own them. Settling on its silence skipped peers that held valid + // rows, and could report `sharedMemoryVerified` with `sharedMemorySynced: 0`. + // + // The durable plane still settles on the curator's round, so the expensive + // half of the walk is still one payload: fallback peers are narrowed to SWM. + const peerIds = Array.from({ length: 6 }, (_, i) => `peer-${i}`); + const durableCalls: string[] = []; + const sharedCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-swm-union', includeSharedMemory: true }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { preferredPeerId: 'peer-0', authoritativePeerId: 'peer-0', isPrivateContextGraph: false, peerIds, connectedPeers: peerIds.length }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + durableCalls.push(args[0] as string); + return durableResult(); + case 'syncSharedMemory': + sharedCalls.push(args[0] as string); + // The curator has nothing; a later member holds the rows. + if (args[0] === 'peer-0') { + return { + ...sharedResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + completedPhases: 2, + emptyResponses: 1, + }; + } + return sharedResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The expensive plane is still pulled once… + expect(durableCalls).toEqual(['peer-0']); + // …while the union plane keeps walking, and reaches the member that has rows. + expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); + expect(result.peersNotAttempted).toBe(0); + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); + expect(result.cleanPlaneCompletions?.sharedMemory.verifiedDataPeers).toBeGreaterThan(0); + expect(result.sharedMemorySynced).toBeGreaterThan(0); }); it('skips the durable plane on fallback peers once the curator settled it', async () => { @@ -619,7 +667,10 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) // …while the unproven shared plane keeps walking every remaining peer. expect([...sharedCalls].sort()).toEqual([...peerIds].sort()); expect(result.peersNotAttempted).toBe(0); - expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(1); + // The shared plane produces no authority evidence at all now — privacy is no + // longer the only thing standing between an empty curator round and a + // settled SWM plane. + expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); }); it('does not settle the SHARED-MEMORY plane on curator metadata alone', async () => { diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index ada021273e..c40ae762d8 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -769,7 +769,7 @@ describe('catch-up plane proof predicates', () => { emptyResponses: 0, }; - it('does not read SHARED-MEMORY metadata as hosting evidence', () => { + it('never reads a SHARED-MEMORY round as hosted-empty evidence', () => { // `/_meta` definition triples are a DURABLE fact: serving them proves // the peer hosts the Context Graph. Shared-memory metadata is a different // artifact, and shared memory is contributed by many members rather than @@ -791,11 +791,13 @@ describe('catch-up plane proof predicates', () => { plane: 'shared-memory', })).toMatchObject({ authorityEmptyPeers: 0 }); - // A genuine wire-empty shared-memory round from the curator still counts. + // Nor does a wire-empty one: on this plane NOBODY's emptiness is + // authoritative, because the curator does not own the members' layers. + // An empty SWM plane is still provable, but only as a whole-round verdict. expect(catchupPeerPlaneEvidence( { ...curatorSharedMetaOnly, insertedTriples: 0, insertedMetaTriples: 0, emptyResponses: 1 }, { fromAuthority: true, plane: 'shared-memory' }, - )).toMatchObject({ authorityEmptyPeers: 1 }); + )).toMatchObject({ authorityEmptyPeers: 0, emptyPeers: 1 }); // …and the identical shape on the DURABLE plane is hosting evidence. expect(catchupPeerPlaneEvidence(curatorSharedMetaOnly, { From 3b9e2ffab2005f0516b048965e7a924e9809ad79 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 15:45:14 +0200 Subject: [PATCH 30/44] fix(sync): fail loudly on the pre-#2006 positional admission shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 20. The finding is correct on the facts: this method took `(…, priorityOverride?: number, operationSignal?: AbortSignal)` positionally and now takes one `admission` object. TypeScript rejects the old shape, but a JS caller compiled against it would pass a number, destructure to `undefined`, and silently lose BOTH its priority override and — worse — its cancellation. An operation that ignores its abort signal keeps running after the caller gave up. Losing cancellation quietly is strictly worse than failing, so the old shape now throws with a message naming the new one. Deliberately NOT the suggested compatibility shim. This repo's standing rule is no compat shims, and the exposure does not justify reversing it: the method has no caller outside `packages/agent` — it is an internal sync-scheduler admission helper that happens to sit on the composed class — so a translated second shape would be carried and tested forever for a caller that does not exist. What the finding correctly identified is the SILENCE, and silence is what this removes. The general question of API breaks on a published 10.0.x line is the same one already open on the `retryDelaysMs` thread for the release owner; this commit does not pre-empt it either way. Mutation: removing the guard kills both new rows. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-lifecycle.ts | 22 ++++++++++++++ .../durable-sync-lifecycle-binding.test.ts | 29 +++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index aa096a652a..2431a59671 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -1185,6 +1185,28 @@ export class LifecycleSyncMethods extends DKGAgentBase { source?: SyncAdmissionSource; } = {}, ): Promise { + // Before #2006 this took `(…, priorityOverride?: number, operationSignal?: AbortSignal)` + // positionally. Those collapsed into one `admission` object so the new `source` + // dimension did not become a fourth positional argument. + // + // TypeScript rejects the old shape, but a JS caller compiled against it would + // pass a number here, destructure to `undefined`, and silently lose BOTH its + // priority override AND its cancellation — an operation that ignores its abort + // signal keeps running after the caller gave up. Losing cancellation quietly is + // strictly worse than failing, so the old shape fails loudly. + // + // Deliberately NOT a compatibility shim translating the old arguments: this is an + // internal admission helper with no caller outside `packages/agent`, and a + // translated second shape would have to be carried and tested forever. + if (typeof admission !== 'object' || admission === null + || typeof (admission as { aborted?: unknown }).aborted === 'boolean') { + throw new TypeError( + 'runContextGraphSyncWithBackpressure takes a single `admission` object ' + + '({ priorityOverride, operationSignal, source }). The positional ' + + 'priority/signal arguments used before issue #2006 are no longer accepted, ' + + 'because ignoring them would silently drop the caller\'s cancellation.', + ); + } const { priorityOverride, operationSignal } = admission; const source = normalizeSyncAdmissionSource(admission.source); const priority = priorityOverride diff --git a/packages/agent/test/durable-sync-lifecycle-binding.test.ts b/packages/agent/test/durable-sync-lifecycle-binding.test.ts index 6a42369a4a..f1fc8cacb5 100644 --- a/packages/agent/test/durable-sync-lifecycle-binding.test.ts +++ b/packages/agent/test/durable-sync-lifecycle-binding.test.ts @@ -344,6 +344,35 @@ describe('durable sync lifecycle chain binding', () => { expect(runLegacyDurableSync.mock.calls[0]?.[6]).not.toHaveProperty('totalTimeoutMs'); }); + it.each([ + ['a positional priority override', 2000], + ['a positional AbortSignal', 'SIGNAL'], + ])('rejects the pre-#2006 positional admission shape: %s', async (_label, sixth) => { + // The old signature was (ctx, cg, lane, label, work, priorityOverride?, signal?). + // A JS caller compiled against it would destructure to undefined and silently + // lose its priority AND its cancellation — an operation that ignores its abort + // signal keeps running after the caller gave up. That must fail loudly. + const sixthArg = sixth === 'SIGNAL' ? new AbortController().signal : sixth; + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + node: { stopSignal: undefined }, + syncScheduler: { acquire: async () => ({ release: () => {} }) }, + }; + + await expect( + (LifecycleSyncMethods.prototype.runContextGraphSyncWithBackpressure as any).call( + agentLike, + {}, + 'cg-legacy', + 'durable', + 'label', + async () => 'done', + sixthArg, + ), + ).rejects.toThrow(/takes a single .admission. object/); + }); + it('labels standalone SWM recovery admissions at the call site', async () => { // The sibling of the VM-recovery assertion above, and the one that had NO // coverage: a regression dropping this source would report SWM recovery From 2e8f38b29c8683efdec05e5d68ffb684138f4d5f Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 16:09:49 +0200 Subject: [PATCH 31/44] fix(sync): reject removed/reshaped options at runtime, not just at compile time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 21. Two findings, one class: a contract this PR changed may BREAK a caller, but it must never DEGRADE one silently. Both confirmed by running code. 1. `retryDelaysMs` was rejected by TypeScript and ignored by the runtime. A JS caller compiled against the pre-#2006 shape passed it and got the wall-clock budget instead of its ladder. Measured against the built dist with an always-deferred run: 41 attempts / 180,000 ms of blocking, where the old ladder gave 3 attempts / 30 ms. It now throws, before the mode branch so a background caller is not exempt. 2. The round-20 guard for the positional admission shape had a hole this found exactly: `(…, work, undefined, signal)`. The 6th argument looks absent and defaults to `{}`, so only the PRESENCE of a 7th reveals a caller that still believes it is passing a cancellation signal. Verified against the built dist — it returned normally and dropped the signal. A `...legacyPositionalArgs: never[]` rest parameter now makes that a compile error AND a runtime throw. Swept the class rather than the two instances. Everything else this PR reshaped is loud or harmless: a removed root export fails at ESM link time; the injected clock/plane callbacks are honoured; `mode`, `includeSharedMemory`, the plane callbacks, `deferredBackpressure` and the priority helpers all still drive the same behaviour. One residual is inherent to JavaScript and not actionable — reading a removed export off a dynamic `import()` namespace yields `undefined` rather than throwing, which is true of every removed export in every package. Each guard mutation-checked in both directions, including the compile-time half: widening the rest parameter to `any[]` fails `test:types`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-lifecycle.ts | 13 ++++++++- packages/agent/src/sync/catchup-policy.ts | 15 +++++++++++ packages/agent/test/catchup-policy.test.ts | 27 +++++++++++++++++++ .../test/catchup-retry-contract.typecheck.ts | 26 ++++++++++++++++++ .../durable-sync-lifecycle-binding.test.ts | 18 +++++++++---- 5 files changed, 93 insertions(+), 6 deletions(-) diff --git a/packages/agent/src/dkg-agent-lifecycle.ts b/packages/agent/src/dkg-agent-lifecycle.ts index 2431a59671..51f29f893c 100644 --- a/packages/agent/src/dkg-agent-lifecycle.ts +++ b/packages/agent/src/dkg-agent-lifecycle.ts @@ -1184,6 +1184,12 @@ export class LifecycleSyncMethods extends DKGAgentBase { */ source?: SyncAdmissionSource; } = {}, + /** + * Nothing may follow `admission`. Typed `never[]` so a TypeScript caller passing + * the old 7th positional `operationSignal` fails to compile, and captured at + * runtime so a JS one fails too — see the guard below. + */ + ...legacyPositionalArgs: never[] ): Promise { // Before #2006 this took `(…, priorityOverride?: number, operationSignal?: AbortSignal)` // positionally. Those collapsed into one `admission` object so the new `source` @@ -1198,7 +1204,12 @@ export class LifecycleSyncMethods extends DKGAgentBase { // Deliberately NOT a compatibility shim translating the old arguments: this is an // internal admission helper with no caller outside `packages/agent`, and a // translated second shape would have to be carried and tested forever. - if (typeof admission !== 'object' || admission === null + // `legacyPositionalArgs` catches the shape the 6th-argument test below cannot: + // `(…, work, undefined, signal)`. There the 6th is absent-looking and defaults to + // `{}`, so only the presence of a 7th argument reveals that a caller still thinks + // it is passing a cancellation signal. + if (legacyPositionalArgs.length > 0 + || typeof admission !== 'object' || admission === null || typeof (admission as { aborted?: unknown }).aborted === 'boolean') { throw new TypeError( 'runContextGraphSyncWithBackpressure takes a single `admission` object ' diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index ae7484b835..ed9399bf74 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -169,6 +169,21 @@ export async function runCatchupPlaneWithPolicy( run: (context: CatchupPlaneContext) => Promise, options: CatchupPlanePolicyClock = {}, ): Promise { + // `retryDelaysMs` configured the fixed [100, 250, 500] ladder that #2006 replaced + // with a wall-clock budget. Retaining it as `?: never` makes a TypeScript caller + // fail to compile — but a JS caller compiled against the old shape still passes it + // and would have it IGNORED, silently turning an intended 10 ms schedule into a wait + // of up to CATCHUP_BACKPRESSURE_MAX_WAIT_MS. Checked before the mode branch so a + // background caller is not exempt. + if ((options as { retryDelaysMs?: unknown }).retryDelaysMs !== undefined) { + throw new TypeError( + 'retryDelaysMs was removed in issue #2006; catch-up retries are now bounded by ' + + 'an absolute wall-clock budget. Use `retry.maxWaitMs` (operators: ' + + 'DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS). It is rejected rather than ignored ' + + 'because ignoring it would silently extend the wait to the full budget.', + ); + } + const context: CatchupPlaneContext = { priority: catchupPriorityForMode(mode), source: catchupSourceForMode(mode), diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index a096efb59d..8539503c87 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -241,6 +241,33 @@ describe('runCatchupPlanesWithPolicy', () => { }); }); +describe('the removed retryDelaysMs ladder', () => { + // Retaining it as `?: never` makes a TypeScript caller fail to compile, which the + // enforced type test pins. But TypeScript is not the runtime: a JS caller compiled + // against the pre-#2006 shape still passes it, and ignoring it would silently turn + // an intended 10 ms schedule into a wait of up to the full budget. So it is + // REJECTED, not ignored. + it.each(['foreground', 'background'] as const)('is rejected at runtime in %s mode', async (mode) => { + await expect( + runCatchupPlaneWithPolicy(mode, async () => ({ deferredBackpressure: 1 }), { + retryDelaysMs: [10, 20], + retry: { maxWaitMs: 50 }, + } as never), + ).rejects.toThrow(/retryDelaysMs was removed/); + }); + + it('leaves the supported replacement working', async () => { + const clock = virtualClock(); + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 300 }, now: clock.now, wait: clock.wait, random: () => 0 }, + ); + expect(result.deferredBackpressure).toBe(1); + expect(clock.elapsed()).toBeLessThanOrEqual(300); + }); +}); + describe('nextCatchupBackpressureDelayMs', () => { it('grows exponentially and clamps at the per-step ceiling', () => { const delays = Array.from({ length: 8 }, (_, attempt) => nextCatchupBackpressureDelayMs({ diff --git a/packages/agent/test/catchup-retry-contract.typecheck.ts b/packages/agent/test/catchup-retry-contract.typecheck.ts index 5640b7758b..5b0363b821 100644 --- a/packages/agent/test/catchup-retry-contract.typecheck.ts +++ b/packages/agent/test/catchup-retry-contract.typecheck.ts @@ -1,5 +1,6 @@ import { CATCHUP_BACKPRESSURE_MAX_WAIT_MS, + DKGAgent, runCatchupPlaneWithPolicy, type CatchupPlanePolicyClock, type CatchupPlanePolicyOptions, @@ -71,3 +72,28 @@ export declare const pinned: [ typeof CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS, typeof runCatchupPlaneWithPolicy, ]; + +// The admission parameters of `runContextGraphSyncWithBackpressure` collapsed from +// positional `(priorityOverride?: number, operationSignal?: AbortSignal)` into a single +// object, so the new `source` dimension did not become a fourth positional argument. +// +// The runtime guard rejects the old shape (see durable-sync-lifecycle-binding.test.ts). +// This pins the COMPILE-TIME half, and specifically the seventh-argument case: without +// the `...legacyPositionalArgs: never[]` rest parameter a stale caller passing a +// trailing AbortSignal type-checks, and TypeScript would say nothing about a caller +// that is quietly losing its cancellation. +declare const staleAdmissionCaller: DKGAgent; +declare const staleSignal: AbortSignal; + +const staleAdmissionCall = () => staleAdmissionCaller.runContextGraphSyncWithBackpressure( + {} as never, + 'cg', + 'durable' as never, + 'label', + async () => 1, + {}, + // @ts-expect-error nothing may follow `admission`; this is the pre-#2006 positional signal. + staleSignal, +); + +export declare const pinnedAdmission: typeof staleAdmissionCall; diff --git a/packages/agent/test/durable-sync-lifecycle-binding.test.ts b/packages/agent/test/durable-sync-lifecycle-binding.test.ts index f1fc8cacb5..51048009ef 100644 --- a/packages/agent/test/durable-sync-lifecycle-binding.test.ts +++ b/packages/agent/test/durable-sync-lifecycle-binding.test.ts @@ -345,14 +345,22 @@ describe('durable sync lifecycle chain binding', () => { }); it.each([ - ['a positional priority override', 2000], - ['a positional AbortSignal', 'SIGNAL'], - ])('rejects the pre-#2006 positional admission shape: %s', async (_label, sixth) => { + ['a positional priority override', [2000]], + ['a positional AbortSignal', ['SIGNAL']], + // The shape the 6th-argument check alone cannot see: the 6th is absent-looking + // and defaults to `{}`, so only the PRESENCE of a 7th reveals that the caller + // still believes it is passing a cancellation signal. Before the rest-parameter + // guard this returned normally and dropped the signal. + ['a cancellation-only legacy call', [undefined, 'SIGNAL']], + ['both legacy positionals', [2000, 'SIGNAL']], + ])('rejects the pre-#2006 positional admission shape: %s', async (_label, tail) => { // The old signature was (ctx, cg, lane, label, work, priorityOverride?, signal?). // A JS caller compiled against it would destructure to undefined and silently // lose its priority AND its cancellation — an operation that ignores its abort // signal keeps running after the caller gave up. That must fail loudly. - const sixthArg = sixth === 'SIGNAL' ? new AbortController().signal : sixth; + const legacyArgs = (tail as unknown[]).map( + (a) => (a === 'SIGNAL' ? new AbortController().signal : a), + ); const agentLike = { config: {}, log: { info: () => {}, warn: () => {}, debug: () => {} }, @@ -368,7 +376,7 @@ describe('durable sync lifecycle chain binding', () => { 'durable', 'label', async () => 'done', - sixthArg, + ...legacyArgs, ), ).rejects.toThrow(/takes a single .admission. object/); }); From 0b7acbf3f4b43802a53865ae4d99d006774c24ea Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 16:18:41 +0200 Subject: [PATCH 32/44] fix(sync): require the plane discriminator instead of defaulting it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 21 (fourth finding). Same class as the other two this round: a silent wrong default rather than a loud one. `plane` was optional and fell back to durable semantics, so a shared-memory call site that forgot it would silently take the durable branch and produce hosted-empty evidence on a plane where no such evidence exists. That is not hypothetical — it is exactly the mutation I ran two commits ago, and only a test noticed it. It is required now. Omitting it at the shared-memory call site is a compile error: error TS2345: Property 'plane' is missing in type '{ fromAuthority: boolean; }' but required in type '{ plane: "durable" | "shared-memory"; … }' Chose a required member over the suggested discriminated union: the union's extra benefit is keeping `complete` off the shared variant, which is a smaller problem than the defaulted plane, at the cost of restructuring every call site and its tests. `complete` is documented as durable-only in place instead. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/cli/src/catchup-runner.ts | 17 +++++++++++------ packages/cli/test/catchup-runner.test.ts | 5 +++-- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index ac75be7f32..bcefe2a78f 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -555,14 +555,19 @@ export function catchupPeerPlaneEvidence( | null | undefined, options: { - complete?: boolean; - fromAuthority?: boolean; /** - * Which plane this result came from. Required for authority evidence - * because "metadata proves the peer hosts the graph" is a DURABLE fact. + * Which plane this result came from. REQUIRED, and deliberately not + * defaulted: the strongest thing this function can say — hosted-empty + * evidence — is true on the durable plane and false on shared memory, so a + * defaulted `plane` would let a shared-memory call site silently take the + * durable branch. Only a test would notice, and the whole point is that a + * mistake here settles a plane nobody proved. */ - plane?: 'durable' | 'shared-memory'; - } = {}, + plane: 'durable' | 'shared-memory'; + /** Durable-only lifecycle state; the shared plane has no `complete` concept. */ + complete?: boolean; + fromAuthority?: boolean; + }, ): CatchupPlaneCompletionEvidence { const none = { verifiedDataPeers: 0, diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index c40ae762d8..9d40ca0ca1 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -809,18 +809,19 @@ describe('catch-up plane proof predicates', () => { it('counts the curator, and ONLY the curator, as hosted-empty evidence', () => { expect(catchupPeerPlaneEvidence(hostedEmptyRound, { + plane: 'durable', complete: true, fromAuthority: true, })).toMatchObject({ verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 1 }); // The identical round from any other peer is the commonest state on the // network — a member holding `_meta` that has not synced the data yet — // and counting it would resettle #2006 as `done` with zero KAs. - expect(catchupPeerPlaneEvidence(hostedEmptyRound, { complete: true })) + expect(catchupPeerPlaneEvidence(hostedEmptyRound, { plane: 'durable', complete: true })) .toMatchObject({ authorityEmptyPeers: 0 }); // Neither does a curator round that fetched data but inserted none. expect(catchupPeerPlaneEvidence( { ...hostedEmptyRound, fetchedDataTriples: 4_000 }, - { complete: true, fromAuthority: true }, + { plane: 'durable', complete: true, fromAuthority: true }, )).toMatchObject({ authorityEmptyPeers: 0 }); }); From d2c77a69b0725b3dfa9eaf488b63af5e4b7541f6 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 16:23:55 +0200 Subject: [PATCH 33/44] test(sync): pin the source handoff from the production helper to the scheduler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 22. The diagnostic contract was covered at both ends and not in the middle: the call sites were proven to SUPPLY a source, and `withGlobalSyncBackpressure` was proven to RENDER `:` — but nothing covered the `source,` line handed between them inside `runContextGraphSyncWithBackpressure`. Deleting it left every existing test green while real admissions would report `durable:unspecified`, losing exactly the trigger attribution issue #2006 had to reconstruct from daemon logs. New test drives the real prototype and reads the shared registry while the admission is active, asserting `durable:catchup-foreground` — and still asserting no Context Graph identifier reaches node-wide diagnostics. Releases the admission in `finally`. The first mutation run exposed why: the registry is shared across the file, so a failed assertion that skipped the release left the operation active and cascaded into four unrelated tests. With the release guaranteed, the mutation kills exactly one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/test/sync-backpressure.test.ts | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/packages/agent/test/sync-backpressure.test.ts b/packages/agent/test/sync-backpressure.test.ts index 8e22e6b980..382fe6b2a0 100644 --- a/packages/agent/test/sync-backpressure.test.ts +++ b/packages/agent/test/sync-backpressure.test.ts @@ -12,6 +12,7 @@ import { withGlobalSyncBackpressure, } from '../src/sync/backpressure.js'; import { PriorityAdmissionQueue } from '../src/sync/priority-admission-queue.js'; +import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; const tick = () => new Promise((resolve) => setTimeout(resolve, 0)); @@ -216,6 +217,53 @@ describe('sync global backpressure', () => { ]); }); + it('carries the admission source from the production helper through to the scheduler', async () => { + // The two halves of this contract were covered separately: the call sites + // were proven to SUPPLY a source, and `withGlobalSyncBackpressure` was proven + // to RENDER it as `:`. Nothing covered the line between + // them — the `source,` handed to `withGlobalSyncBackpressure` inside + // `runContextGraphSyncWithBackpressure`. Dropping it leaves both groups green + // while every real admission reports `durable:unspecified` on + // /api/diagnostics/backpressure, which is the attribution issue #2006 had to + // reconstruct from daemon logs. + const agentLike = { + config: { syncGlobalMaxInflight: 1, syncGlobalQueueLimit: 1 }, + node: { stopSignal: undefined }, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + }; + + let releaseWork!: () => void; + const admitted = LifecycleSyncMethods.prototype.runContextGraphSyncWithBackpressure.call( + agentLike as never, + createOperationContext('sync'), + 'urn:cg:private:e2e', + 'durable' as never, + 'durable:urn:cg:private:e2e', + () => new Promise((resolve) => { releaseWork = resolve; }), + { source: 'catchup-foreground' }, + ); + await tick(); + + // Release in `finally`: this admission is registered in the SHARED + // backpressureRegistry, so a failed assertion that skipped the release would + // leave it active and cascade into every later test in this file. + try { + const snapshot = backpressureRegistry.capture().schedulers.find( + (scheduler) => scheduler.scheduler === 'sync-global', + ); + expect(snapshot).toMatchObject({ + lanes: [expect.objectContaining({ + activeOperations: [expect.objectContaining({ operation: 'durable:catchup-foreground' })], + })], + }); + // …and the Context Graph id still never reaches node-wide diagnostics. + expect(JSON.stringify(snapshot)).not.toContain('urn:cg:private'); + } finally { + releaseWork(); + await admitted; + } + }); + it('removes CG and peer correlation identifiers from node-wide pressure diagnostics', async () => { const ctx = createOperationContext('sync'); const policy = resolveSyncGlobalBackpressure({ From cf72f24044a600d1abbed7086d1fe7dde6937be6 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 16:31:06 +0200 Subject: [PATCH 34/44] fix(sync): an injected `wait` without `now` spins for the whole budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by sweeping the defect class rather than the reported instances — no reviewer raised this one, and it is the worst of the four. `wait` and `now` are ONE seam. Before #2006 the retry loop was bounded by the fixed `retryDelaysMs` ladder, so an injected `wait` that resolved instantly still terminated after three steps. #2006 deleted the ladder and moved the terminator onto a wall-clock deadline read through `now` — so a caller injecting only `wait` lost its bound entirely: `wait` resolves immediately while `now` stays the real clock, and the loop spins as fast as the microtask queue allows for the ENTIRE budget. Measured against the built module: 6,873,671 attempts in a 2 s budget with the macrotask queue starved throughout — roughly 700 million at the shipped 180 s default. With a frozen `now` it never terminates at all. Unlike the other three, this is not only a stale-caller hazard: `{ wait }` alone is what a NEW caller naturally writes to keep a test fast, and it type-checks today. Rejected rather than defaulted — silently pairing it with the real clock IS the hang, and silently pairing it with a fake one would invent a timeline the caller never asked for. Placed after the background early-return, since background mode never enters the loop and one test legitimately injects `wait` alone to assert that. Both the guard and its placement are mutation-pinned, alongside a positive test that the un-injected production path — the only shape the agent and CLI worker use — is untouched. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/sync/catchup-policy.ts | 30 +++++++++++++ packages/agent/test/catchup-policy.test.ts | 51 ++++++++++++++++++++++ 2 files changed, 81 insertions(+) diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index ed9399bf74..4408379700 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -190,6 +190,36 @@ export async function runCatchupPlaneWithPolicy( }; if (mode !== 'foreground') return run(context); + // `wait` and `now` are ONE seam, not two independent ones. + // + // Before #2006 the loop was bounded by the fixed `retryDelaysMs` ladder, so an + // injected `wait` could return immediately and the loop still ended after three + // steps. #2006 deleted the ladder and moved the terminator onto a wall-clock + // deadline read through `now` — so a caller that injects only `wait` no longer + // has a bound: `wait` resolves instantly while `now` is the real clock, and the + // loop spins as fast as the microtask queue allows for the WHOLE budget. + // Measured against the built module: 6,873,671 attempts in a 2 s budget, with + // the macrotask queue starved throughout — roughly 700 million at the shipped + // 180 s default. A frozen `now` never terminates at all. + // + // This is not only a stale-caller hazard: `{ wait }` alone is what a NEW caller + // naturally writes to keep a test fast, and it type-checks today. Rejected here + // rather than defaulted, because silently pairing it with the real clock is the + // hang, and silently pairing it with a fake one would invent a timeline the + // caller never asked for. + // + // Placed after the background early-return on purpose: background mode never + // enters the retry loop, so injecting `wait` alone there is harmless and one + // test legitimately does it to assert the loop is not entered. + if (options.wait !== undefined && options.now === undefined) { + throw new TypeError( + 'runCatchupPlaneWithPolicy: `wait` and `now` must be injected together. ' + + 'Since issue #2006 the retry loop is bounded by a wall-clock deadline read ' + + 'through `now`, so an injected `wait` without a matching `now` spins for the ' + + 'entire budget instead of stepping a schedule.', + ); + } + const now = options.now ?? Date.now; const wait = options.wait ?? defaultWait; const maxWaitMs = options.retry?.maxWaitMs ?? CATCHUP_BACKPRESSURE_MAX_WAIT_MS; diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index 8539503c87..13248f1ceb 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -241,6 +241,57 @@ describe('runCatchupPlanesWithPolicy', () => { }); }); +describe('the wait/now clock seam', () => { + // Before #2006 `retryDelaysMs` bounded the loop, so an injected `wait` that + // resolved instantly still terminated after three steps. The ladder is gone and + // the terminator now lives behind `now`, so injecting `wait` ALONE removes the + // bound: measured against the built module, 6,873,671 attempts in a 2 s budget + // with the macrotask queue starved — ~700 million at the shipped 180 s default. + // `{ wait }` on its own is also what a NEW caller naturally writes to keep a + // test fast, so this is a live footgun, not only a stale-caller hazard. + it('rejects an injected wait with no matching now', async () => { + await expect( + runCatchupPlaneWithPolicy('foreground', async () => ({ deferredBackpressure: 1 }), { + wait: async () => {}, + retry: { maxWaitMs: 50 }, + }), + ).rejects.toThrow(/must be injected together/); + }); + + it('allows an injected wait in background mode, which never enters the loop', async () => { + // The complement, so the guard cannot be widened into something that breaks a + // legitimate caller: background mode returns before the retry loop, so `wait` + // alone is harmless there and one test below relies on exactly that. + await expect( + runCatchupPlaneWithPolicy('background', async () => ({ deferredBackpressure: 1 }), { + wait: async () => { throw new Error('background mode must not wait'); }, + }), + ).resolves.toEqual({ deferredBackpressure: 1 }); + }); + + it('allows the paired seam, and it still terminates on the budget', async () => { + const clock = virtualClock(); + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 300 }, now: clock.now, wait: clock.wait, random: () => 0 }, + ); + expect(result.deferredBackpressure).toBe(1); + expect(clock.elapsed()).toBeLessThanOrEqual(300); + }); + + it('leaves the un-injected production path alone', async () => { + // Neither the agent nor the CLI worker injects a clock, so the default + // `Date.now` + real `setTimeout` pairing must keep working untouched. + const result = await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 0 } }, + ); + expect(result.deferredBackpressure).toBe(1); + }); +}); + describe('the removed retryDelaysMs ladder', () => { // Retaining it as `?: never` makes a TypeScript caller fail to compile, which the // enforced type test pins. But TypeScript is not the runtime: a JS caller compiled From be6da3fca49453b91b3ac4efb95fa41c86ba28f3 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 16:50:31 +0200 Subject: [PATCH 35/44] fix(sync): the walk's stop rule must consult the round, like readiness does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 23. I had claimed the walk and the readiness classifier "cannot drift apart" because both consume `authorityEmptyPeers`. They could: readiness voids a hosted-empty proof when the ROUND contradicts it — data fetched, KCs rejected, data rejected for missing metadata — and the walk's copy of the rule looked only at the curator's own response. So the walk could stop on a curator whose word the round had already contradicted, skip peers that might hold valid content, and then report the job unready. The worst of both. The walk now calls `catchupPlaneProvenByAuthorityHostedEmpty` itself, with the round's diagnostics — the same function, not a restatement of it. The curator's own evidence is tracked separately from the round total, because proof-by-data must come from the curator alone while the contradiction is round-wide. Settled at the END of a wave rather than per peer, so a contradiction raised by any member of the same wave is visible regardless of the order results arrived. Also pins the worker `error` latch, which moved from a one-off pending rejection to the shared `fail()` and was covered only through `exit`. Restoring the old behaviour let a LATER subscribe post into a dead worker — the half of #2006's hang that made every subsequent job stick at `running`. Both mutation-checked. The first attempt at the contradiction test did NOT kill its mutant: with three peers the opening wave was full width, so every peer was contacted regardless and no early stop could be observed. The fixture now puts the surviving peer behind a real wave boundary. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../cli/src/catchup-runner-worker-impl.ts | 79 +++++++++++------- .../test/catchup-runner-worker-impl.test.ts | 83 +++++++++++++++++++ .../catchup-runner-worker-lifecycle.test.ts | 33 ++++++++ 3 files changed, 165 insertions(+), 30 deletions(-) diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 065ee8b331..c9aee06bdc 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -16,6 +16,7 @@ import { catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, + catchupPlaneProvenByAuthorityHostedEmpty, catchupPlaneProvenByData, type CatchupJobResult, type CatchupPlaneCompletionEvidence, @@ -214,22 +215,56 @@ async function runCatchup(request: CatchupRunRequest): Promise const authorityProvedEverything = (): boolean => authorityProven.durable && (!request.includeSharedMemory || authorityProven.sharedMemory); + /** + * The CURATOR's own evidence, kept apart from the round total. + * + * The round total mixes in every peer, and only the curator may end the walk — + * so proof-by-data has to be read from this, not from + * `cleanPlaneCompletions`, or any peer's data would stop it. + */ + const authorityEvidence: Record<'durable' | 'sharedMemory', CatchupPlaneCompletionEvidence> = { + durable: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, + sharedMemory: { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, + }; + /** * Whether the curator's round settles a plane well enough to stop walking. * - * Verified content always does. A content-free round only does for a PUBLIC - * graph: readiness deliberately refuses to prove a private plane from an - * empty response, so stopping on one would strand the walk without proving - * anything — skipping fallback peers that may hold authorized private data - * and turning a recoverable catch-up into `unreachable`. A verified - * private-only response is content, not emptiness, and still counts. + * Verified content from the curator always does. Its EMPTINESS is weaker: it + * is exactly `catchupPlaneProvenByAuthorityHostedEmpty`, the same predicate + * the readiness classifier applies — called here with the round's diagnostics + * so the two cannot disagree. + * + * That matters, because the round can contradict the curator. If another peer + * fetched data, or served content that failed verification, the curator's + * "there is nothing here" is stale and readiness voids it. Stopping the walk + * on it anyway would skip peers that might have delivered valid content and + * then report the job unready — the worst of both. * - * `authorityEmptyPeers` is set by the same reducer readiness consumes, so the - * stop condition and the readiness verdict cannot drift apart. + * Evaluated at the END of a wave rather than per peer, so a contradiction + * raised by ANY member of the same wave is already visible regardless of the + * order results happened to arrive in. */ - const authoritySettles = (evidence: CatchupPlaneCompletionEvidence): boolean => - catchupPlaneProvenByData(evidence) - || (!prepared.isPrivateContextGraph && (evidence.authorityEmptyPeers ?? 0) > 0); + const authoritySettles = ( + plane: 'durable' | 'sharedMemory', + ): boolean => catchupPlaneProvenByData(authorityEvidence[plane]) + || catchupPlaneProvenByAuthorityHostedEmpty( + authorityEvidence[plane], + diagnostics[plane], + { isPrivate: prepared.isPrivateContextGraph }, + ); + + /** Fold the wave's accumulated state into the stop flags. */ + const settleAuthorityForWave = (): void => { + if (!authorityProven.durable && authoritySettles('durable')) { + authorityProven.durable = true; + } + if (request.includeSharedMemory + && !authorityProven.sharedMemory + && authoritySettles('sharedMemory')) { + authorityProven.sharedMemory = true; + } + }; // Isolate per-peer failures: if one peer's sync steps throw, aggregate what we // can from the other peers instead of failing the entire subscribe/catch-up. @@ -310,17 +345,7 @@ async function runCatchup(request: CatchupRunRequest): Promise plane: 'durable', }); addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); - // The curator answering cleanly settles this plane whether it carried - // data or was legitimately empty: "the host says there is nothing here" - // is the authoritative empty proof, and without it a graph with no - // public data on one plane could never stop the walk. - // - // The positive half runs through the SAME predicate the readiness - // classifier uses, just applied to one peer's evidence rather than the - // round's, so the stop condition cannot drift from the readiness rule. - if (fromAuthority && authoritySettles(durableEvidence)) { - authorityProven.durable = true; - } + if (fromAuthority) addCatchupPlaneEvidence(authorityEvidence.durable, durableEvidence); } if (shared) { @@ -348,14 +373,7 @@ async function runCatchup(request: CatchupRunRequest): Promise // evidence only ever has data/empty set — the same reducer still applies. const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority, plane: 'shared-memory' }); addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); - // Same rule as durable: the curator settles the plane by answering - // cleanly, with data or empty. Shared memory is frequently empty for a - // graph that has durable data, and `includeSharedMemory` defaults to - // true on subscribe, so without this the early stop would almost never - // fire in the shape the fix targets. - if (fromAuthority && authoritySettles(sharedEvidence)) { - authorityProven.sharedMemory = true; - } + if (fromAuthority) addCatchupPlaneEvidence(authorityEvidence.sharedMemory, sharedEvidence); } if (peerDenied) { @@ -440,6 +458,7 @@ async function runCatchup(request: CatchupRunRequest): Promise syncPeer, ); for (const round of rounds) accumulate(round); + settleAuthorityForWave(); if (CATCHUP_STOP_ON_PROOF && authorityProvedEverything()) break; } diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 614dac4318..e7ed6aa7c9 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -673,6 +673,89 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); }); + it('does not stop on a hosted-empty curator that the round already contradicted', async () => { + // The curator says "nothing here" while another peer in the SAME wave served + // content that failed verification. Readiness treats that as content + // EXISTING, so it voids the empty proof — and if the walk had already + // stopped on the curator's word, the job ends unready having skipped peers + // that might have delivered valid data. Worst of both. + // + // The curator is deliberately NOT first, so the opening wave is full width + // and both responses land in the same wave: this is exactly the ordering + // where a per-peer stop decision cannot see the contradiction. + // `peer-later` MUST sit in a later wave: with the curator not first the + // opening wave is full width, so a three-peer list would contact everyone + // regardless and the test could not observe an early stop at all. + const wave1 = ['peer-rejected', 'peer-curator', 'peer-quiet-a', 'peer-quiet-b']; + const peerIds = [...wave1, 'peer-later', 'peer-quiet-c']; + expect(wave1.length).toBe(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + expect(peerIds.length).toBeGreaterThan(CATCHUP_MAX_CONCURRENT_PEER_SYNCS); + const durableCalls: string[] = []; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-contradicted', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + durableCalls.push(args[0] as string); + if (args[0] === 'peer-curator') { + return { + ...durableResult(), + insertedTriples: 9, + fetchedMetaTriples: 9, + fetchedDataTriples: 0, + insertedMetaTriples: 9, + insertedDataTriples: 0, + metaOnlyResponses: 1, + completedPhases: 2, + }; + } + if (args[0] === 'peer-rejected') { + // Served content for this graph; verification threw it out. + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 4_000, + insertedDataTriples: 0, + rejectedKcs: 1, + }; + } + if (args[0] === 'peer-later') return durableResult(); + // Everyone else answers content-free, so the only verified data in the + // run is the one behind the wave boundary. + return { + ...durableResult(), + insertedTriples: 0, + fetchedDataTriples: 0, + insertedDataTriples: 0, + bytesReceived: 0, + emptyResponses: 1, + completedPhases: 2, + }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + // The contradiction is visible round-wide, so the curator's emptiness does + // not settle the plane and the remaining peer is still reached. + expect(durableCalls).toContain('peer-later'); + expect(result.peersNotAttempted).toBe(0); + // …and that last peer's verified data is what actually proves the plane. + expect(result.cleanPlaneCompletions?.durable.verifiedDataPeers).toBeGreaterThan(0); + }); + it('does not settle the SHARED-MEMORY plane on curator metadata alone', async () => { // End-to-end counterpart of the plane-aware reducer: shared memory is // contributed by many members rather than owned by the curator, so diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index eee243188d..4b14ac7002 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -38,6 +38,11 @@ const workerControl = vi.hoisted(() => { for (const listener of state.listeners.get('message') ?? []) listener(message); } + /** Test-only: raise an arbitrary worker event (`error`, `exit`, …). */ + static emitEvent(event: string, ...args: unknown[]) { + for (const listener of state.listeners.get(event) ?? []) listener(...args); + } + postMessage(message: unknown) { state.posted.push(message); } @@ -105,6 +110,34 @@ describe('WorkerCatchupRunner lifecycle', () => { expect(workerControl.state.posted).toHaveLength(postedBeforeLaterRun); }); + it('latches a worker `error` for the in-flight run AND every later one', async () => { + // Node usually emits `exit` after `error`, so the exit tests cover many real + // crashes indirectly — but the `error` handler moved from a one-off pending + // rejection to the shared latch, and nothing pinned that. Restoring the old + // behaviour would let a LATER subscribe post into a dead worker again, which + // is the half of #2006's hang that made every subsequent job stick at + // `running`. + const runner = createCatchupRunner(stubAgent); + const inFlight = runner.run({ contextGraphId: 'cg-crash', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + expect(workerControl.state.posted).toHaveLength(1); + + workerControl.FakeWorker.emitEvent('error', new Error('boom')); + + const first = await withinTick(inFlight); + expect(first).toBeInstanceOf(Error); + expect((first as Error).message).toContain('boom'); + + // The latch, not just this run: a later run must fail fast… + const postedBeforeLater = workerControl.state.posted.length; + const later = runner.run({ contextGraphId: 'cg-after-crash', includeSharedMemory: false }) + .then(() => 'resolved' as const, (error: Error) => error); + const second = await withinTick(later); + expect(second).toBeInstanceOf(Error); + // …and must not have queued work onto the dead worker. + expect(workerControl.state.posted).toHaveLength(postedBeforeLater); + }); + it('rejects every pending run exactly once', async () => { const runner = createCatchupRunner(stubAgent); const first = runner.run({ contextGraphId: 'cg-a', includeSharedMemory: false }) From b649b6775ed72014f16bad821b83b93dd5a80a46 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 17:47:39 +0200 Subject: [PATCH 36/44] fix(sync): a silent CURATOR voids the empty verdict; a silent stranger does not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 24. The reported bug is real and reproduces end to end: one unrelated stranger's empty answer settled `jobStatus: done` with `synced: true` while the peer that may have held the graph was never reached. It is produced by the walk's own design — a resolvable curator gets wave 1 alone, so when it transport-fails the walk moves on to strangers. The obvious fix — void on `failedPeers` — is WRONG, and I built and rejected it before shipping this one. `failedPeers` counts any unreachable peer, so it also kills the verdict when no curator is resolvable at all, which is exactly the state where the hosted-empty backstop structurally cannot fire. A single unreachable stranger would then pin a legitimately empty public graph at `unreachable` — the liveness failure this rule was originally written to avoid, and worse on the shared-memory plane, which since round 19 has no hosted-empty backstop by design. Scoped to the AUTHORITY instead. `authorityUnanswered` records that a resolvable curator was selected and never cleanly answered THIS plane; only that voids the whole-round empty verdict. An unreachable stranger is still evidence of nothing. Also in this round: - `CatchupAdmissionSource` is now derived from `SYNC_ADMISSION_SOURCES` via `Extract` rather than restating its literals, so renaming a member there is a compile error instead of a stale label the scheduler clamps to `unspecified`. - The changelog delta lane's admission source is pinned at its call site; it was a third production admission path with no coverage. - CHANGELOG records that node operators need do nothing for the `retryDelaysMs` removal, since the node ships as one unit (`workspace:*`) — the removal is only visible to code outside this repo installing the agent from npm. Every branch mutation-checked, including the two liveness cases the naive fix would have broken and the worker-side production of the flag. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- CHANGELOG.md | 4 +- packages/agent/src/sync/catchup-policy.ts | 17 ++++- .../durable-sync-lifecycle-binding.test.ts | 43 ++++++++++++ .../cli/src/catchup-runner-worker-impl.ts | 36 +++++++++- packages/cli/src/catchup-runner.ts | 34 +++++++++ .../test/catchup-runner-worker-impl.test.ts | 70 +++++++++++++++++++ packages/cli/test/catchup-runner.test.ts | 39 +++++++++++ 7 files changed, 238 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a8517811d5..4dc16108ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,9 @@ All notable changes to the DKG V10 node are documented here. The format is based ### Removed -- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. `retryDelaysMs` is retained on the options type as `never`, so a caller that still sets it fails to compile rather than having it silently ignored — an ignored `retryDelaysMs: [10]` would otherwise turn an intended 10 ms schedule into a wait of up to the full budget. +- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. `retryDelaysMs` is retained on the options type as `never`, so a caller that still sets it fails to compile rather than having it silently ignored, and both it and the removed export are rejected at RUNTIME too — an ignored `retryDelaysMs: [10]` would otherwise turn an intended 10 ms schedule into a wait of up to the full budget, measured at 41 retry attempts and 180,000 ms of blocking against the old ladder's 3 attempts and 30 ms. + + **Node operators need do nothing.** The node ships as one unit — the CLI depends on the agent as `workspace:*`, so every package moves to the same version on upgrade and no node holds a stale caller. This removal is only visible to code OUTSIDE this repository that installs `@origintrail-official/dkg-agent` from npm and calls the catch-up retry policy directly, which is an internal sync-scheduler knob rather than part of the SDK surface. Anything that does hit it gets an immediate error naming the replacement, not a silent behaviour change. ### Operator knobs diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 4408379700..7139ba5557 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -1,3 +1,5 @@ +import type { SyncAdmissionSource } from './policy.js'; + export type CatchupMode = 'background' | 'foreground'; export const FOREGROUND_CATCHUP_SYNC_PRIORITY = 2_000; @@ -50,8 +52,19 @@ export function resolveCatchupBackpressureMaxWaitMs(raw: string | undefined): nu export const CATCHUP_BACKPRESSURE_MAX_WAIT_MS: number = resolveCatchupBackpressureMaxWaitMs(process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); -/** Bounded admission origin recorded on node-wide scheduler diagnostics. */ -export type CatchupAdmissionSource = 'catchup-foreground' | 'catchup-background'; +/** + * Bounded admission origin recorded on node-wide scheduler diagnostics. + * + * DERIVED from the scheduler's own closed set rather than restating its literals. + * The boundedness of this label space is load-bearing — it is a metric and log + * dimension — and the scheduler owns that contract. Restating the strings meant a + * rename in `SYNC_ADMISSION_SOURCES` would leave `catchupSourceForMode` returning a + * stale literal that the scheduler then silently clamped to `unspecified`, with + * nothing connecting the two declarations. `Extract` makes that a compile error: + * a renamed member collapses this to `never` and the returns below stop building. + */ +export type CatchupAdmissionSource = + Extract; export interface CatchupPlaneResult { deferredBackpressure?: number; diff --git a/packages/agent/test/durable-sync-lifecycle-binding.test.ts b/packages/agent/test/durable-sync-lifecycle-binding.test.ts index 51048009ef..2a4eec0c6a 100644 --- a/packages/agent/test/durable-sync-lifecycle-binding.test.ts +++ b/packages/agent/test/durable-sync-lifecycle-binding.test.ts @@ -18,6 +18,8 @@ vi.mock('../src/sync/requester/graph-scoped-materialization.js', async (importOr }; }); +import { PROTOCOL_SYNC_CHANGELOG } from '@origintrail-official/dkg-core'; +import { createDurableSyncAccumulator } from '../src/sync/durable-progress.js'; import { DKGAgent } from '../src/dkg-agent.js'; import { LifecycleSyncMethods } from '../src/dkg-agent-lifecycle.js'; import { @@ -381,6 +383,47 @@ describe('durable sync lifecycle chain binding', () => { ).rejects.toThrow(/takes a single .admission. object/); }); + it('labels changelog-lane admissions at the call site', async () => { + // The changelog delta lane (OT-RFC-59) is a SEPARATE production admission path + // from the durable and shared-memory ones already covered. A public Context + // Graph on a changelog-capable peer never reaches `runLegacyDurableSync`, so a + // dropped source here would surface only as `changelog:unspecified` on + // /api/diagnostics/backpressure while every existing source test stayed green. + const admissions: unknown[][] = []; + const agentLike = { + config: {}, + log: { info: () => {}, warn: () => {}, debug: () => {} }, + getPeerProtocols: async () => [PROTOCOL_SYNC_CHANGELOG], + isPrivateContextGraph: async () => false, + // Record the admission and return a real accumulator: the lane folds the + // result, so an empty object would fail inside the merge before the + // assertion below could run. + runContextGraphSyncWithBackpressure: async (...args: unknown[]) => { + admissions.push(args); + return createDurableSyncAccumulator(); + }, + }; + + await LifecycleSyncMethods.prototype.runChangelogLane.call( + agentLike as never, + ctx, + '12D3KooWChangelogPeer', + ['public-cg'], + undefined, + 2_000, + 'catchup-foreground', + ); + + expect(admissions).toHaveLength(1); + const [, contextGraphId, lane, , , admission] = admissions[0] as unknown[]; + expect(contextGraphId).toBe('public-cg'); + expect(lane).toBe('changelog'); + expect(admission).toMatchObject({ + priorityOverride: 2_000, + source: 'catchup-foreground', + }); + }); + it('labels standalone SWM recovery admissions at the call site', async () => { // The sibling of the VM-recovery assertion above, and the one that had NO // coverage: a regression dropping this source would report SWM recovery diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index c9aee06bdc..3bbe9ab32d 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -16,6 +16,7 @@ import { catchupPeerPlaneEvidence, catchupPeerResponded, catchupPeerSucceeded, + catchupPlaneCompletedWithoutFailure, catchupPlaneProvenByAuthorityHostedEmpty, catchupPlaneProvenByData, type CatchupJobResult, @@ -254,6 +255,17 @@ async function runCatchup(request: CatchupRunRequest): Promise { isPrivate: prepared.isPrivateContextGraph }, ); + /** + * Did the curator cleanly answer this plane at all? + * + * Separate from `authorityProven`: a curator that answered with data proves the + * plane, and one that answered content-free may or may not, but BOTH count as + * having answered. What readiness needs to know is the third case — the curator + * was selected and we never heard a clean word from it — because then a + * stranger's empty response cannot stand for the graph. + */ + const authorityAnswered = { durable: false, sharedMemory: false }; + /** Fold the wave's accumulated state into the stop flags. */ const settleAuthorityForWave = (): void => { if (!authorityProven.durable && authoritySettles('durable')) { @@ -345,7 +357,12 @@ async function runCatchup(request: CatchupRunRequest): Promise plane: 'durable', }); addCatchupPlaneEvidence(cleanPlaneCompletions.durable, durableEvidence); - if (fromAuthority) addCatchupPlaneEvidence(authorityEvidence.durable, durableEvidence); + if (fromAuthority) { + addCatchupPlaneEvidence(authorityEvidence.durable, durableEvidence); + if (catchupPlaneCompletedWithoutFailure(durable, durable.complete)) { + authorityAnswered.durable = true; + } + } } if (shared) { @@ -373,7 +390,12 @@ async function runCatchup(request: CatchupRunRequest): Promise // evidence only ever has data/empty set — the same reducer still applies. const sharedEvidence = catchupPeerPlaneEvidence(shared, { fromAuthority, plane: 'shared-memory' }); addCatchupPlaneEvidence(cleanPlaneCompletions.sharedMemory, sharedEvidence); - if (fromAuthority) addCatchupPlaneEvidence(authorityEvidence.sharedMemory, sharedEvidence); + if (fromAuthority) { + addCatchupPlaneEvidence(authorityEvidence.sharedMemory, sharedEvidence); + if (catchupPlaneCompletedWithoutFailure(shared)) { + authorityAnswered.sharedMemory = true; + } + } } if (peerDenied) { @@ -462,6 +484,16 @@ async function runCatchup(request: CatchupRunRequest): Promise if (CATCHUP_STOP_ON_PROOF && authorityProvedEverything()) break; } + // A curator we resolved but never heard cleanly from makes the round + // incomplete rather than empty. Recorded per plane, since a curator can answer + // one and fail the other. + if (prepared.authoritativePeerId !== undefined) { + diagnostics.durable.authorityUnanswered = !authorityAnswered.durable; + if (request.includeSharedMemory) { + diagnostics.sharedMemory.authorityUnanswered = !authorityAnswered.sharedMemory; + } + } + diagnostics.noProtocolPeers = noProtocolPeers; if (deferredBackpressure === 0) { await invoke('finalizeCatchup', request.contextGraphId, dataSynced, sharedMemorySynced); diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index bcefe2a78f..1087e21671 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -86,6 +86,9 @@ export interface CatchupJobResult { failedPhases: number; deferredBackpressure: number; deniedPhases?: number; + /** A resolvable curator never cleanly answered this plane; see + * `catchupPlaneProvenByUnanimousEmpty`. */ + authorityUnanswered?: boolean; }; sharedMemory: { fetchedMetaTriples: number; @@ -103,6 +106,9 @@ export interface CatchupJobResult { failedPhases: number; deferredBackpressure: number; deniedPhases?: number; + /** A resolvable curator never cleanly answered this plane; see + * `catchupPlaneProvenByUnanimousEmpty`. */ + authorityUnanswered?: boolean; }; }; } @@ -536,6 +542,12 @@ export interface CatchupPlaneRoundDiagnostics { /** Durable-only integrity rejections; the shared-memory plane never sets them. */ dataRejectedMissingMeta?: number; rejectedKcs?: number; + /** + * A metadata-resolved curator WAS selected for this walk and did not cleanly + * answer this plane — it transport-failed, timed out, was denied, or never got + * contacted. Distinct from `failedPeers`, which counts any unreachable peer. + */ + authorityUnanswered?: boolean; } /** @@ -721,8 +733,27 @@ export function catchupPlaneProvenByAuthorityHostedEmpty( * proof mode 1 has already settled the plane, so voiding here costs the * legitimately-empty graph nothing. * + * The verdict IS voided when the round had a resolvable curator that never + * cleanly answered (`authorityUnanswered`). The peer best placed to know is the + * one we failed to hear from, so "nobody had anything" is not established — the + * round is incomplete, not empty. That closes issue #2006's own symptom in its + * sharpest form: the walk puts a resolvable curator alone in wave 1, so when the + * curator transport-fails the walk moves on to strangers, one answers empty, and + * 40 Knowledge Assets get reported as zero. + * + * Scoped to the AUTHORITY rather than to `failedPeers`, and the difference is + * load-bearing. `failedPeers` counts any unreachable peer, so voiding on it would + * also kill the verdict when NO curator is resolvable at all — the state where + * the hosted-empty backstop structurally cannot fire — leaving a legitimately + * empty public graph pinned at `unreachable` by a single unreachable stranger. + * That is the liveness failure this rule was originally written to avoid, and it + * is still worth avoiding; it is only the curator's silence that is decisive. + * * Two counters are deliberately NOT consulted: * + * - `failedPeers`. A transport failure to a peer we never heard from, which on a + * live testnet can be most of the connected set. An unreachable STRANGER is + * evidence of nothing; an unreachable CURATOR is, and has its own signal above. * - `fetchedMetaTriples`. A raw triple count, not a per-peer verdict: a delta * sync legitimately carries the whole metadata phase with nothing newer than * the watermark, and the requester deliberately does NOT flag that as @@ -753,6 +784,9 @@ export function catchupPlaneProvenByUnanimousEmpty( const cleanEmptyObserved = (completion?.emptyPeers ?? 0) > 0 || (diagnostics?.emptyResponses ?? 0) > 0; if (!cleanEmptyObserved) return false; + // The one peer whose silence is decisive. See the note above for why this is + // scoped to the curator rather than to `failedPeers`. + if (diagnostics?.authorityUnanswered) return false; return (diagnostics?.failedPhases ?? 0) === 0 && (diagnostics?.timedOutPhases ?? 0) === 0 && (diagnostics?.deniedPhases ?? 0) === 0 diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index e7ed6aa7c9..78a3c88e6b 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -673,6 +673,76 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(result.cleanPlaneCompletions?.sharedMemory.authorityEmptyPeers).toBe(0); }); + it('reports the curator as unanswered when it was selected and transport-failed', async () => { + // The exact #2006 shape, produced by the walk's own design: a resolvable + // curator is ranked first and gets wave 1 ALONE, so when it transport-fails + // the walk moves on to strangers, one answers empty, and without this signal + // that stranger's silence would settle a 40-KA graph as `done` with zero. + const peerIds = ['peer-curator', 'peer-a', 'peer-b', 'peer-c']; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-curator-silent', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + if (args[0] === 'peer-curator') { + // Transport failure: no clean completion from the one peer that knows. + return { ...durableResult(), complete: false, insertedTriples: 0, + fetchedDataTriples: 0, insertedDataTriples: 0, bytesReceived: 0, + completedPhases: 0, failedPeers: 1 }; + } + return { ...durableResult(), insertedTriples: 0, fetchedDataTriples: 0, + insertedDataTriples: 0, bytesReceived: 0, completedPhases: 2, emptyResponses: 1 }; + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(result.diagnostics?.durable.authorityUnanswered).toBe(true); + // Strangers still answered cleanly empty — that is exactly what must NOT + // settle the plane now. + expect(result.cleanPlaneCompletions?.durable.emptyPeers).toBeGreaterThan(0); + }); + + it('reports the curator as answered when it completed cleanly', async () => { + // The complement, so the flag cannot be hardwired true: a curator that + // answers must leave the round provable. + const peerIds = ['peer-curator', 'peer-a']; + + const result = await runWorkerCatchup({ contextGraphId: 'cg-curator-answered', includeSharedMemory: false }, async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: 'peer-curator', + authoritativePeerId: 'peer-curator', + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': + return durableResult(); + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }); + + expect(result.diagnostics?.durable.authorityUnanswered).toBe(false); + }); + it('does not stop on a hosted-empty curator that the round already contradicted', async () => { // The curator says "nothing here" while another peer in the SAME wave served // content that failed verification. Readiness treats that as content diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 9d40ca0ca1..1ce98c800a 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -918,6 +918,45 @@ describe('catch-up plane proof predicates', () => { }); }); + describe('a curator that was selected but never cleanly answered', () => { + // The walk puts a resolvable curator ALONE in wave 1, so when it + // transport-fails the walk moves on to strangers, one answers empty, and the + // graph's 40 Knowledge Assets get reported as zero. That is issue #2006's own + // symptom in its sharpest form. + const curatorSilent = { ...cleanEmptyRound, failedPeers: 1, authorityUnanswered: true }; + + it('cannot have its plane proven by a stranger answering empty', () => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, curatorSilent, { isPrivate: false })) + .toBe(false); + expect(catchupPlaneReady(emptyPeers, curatorSilent, { isPrivate: false })).toBe(false); + }); + + it.each([ + // No curator resolved at all. The hosted-empty backstop structurally + // cannot fire here, so voiding on a mere unreachable STRANGER would pin a + // legitimately empty public graph at `unreachable` forever — the liveness + // failure this rule exists to avoid. Only the CURATOR's silence is decisive. + ['no curator was resolvable', { ...cleanEmptyRound, failedPeers: 4 }], + // Registered-but-empty public graph on a lossy network, curator absent + // from the round entirely. + ['the graph is registered but empty', { + ...cleanEmptyRound, fetchedMetaTriples: 9, emptyResponses: 3, failedPeers: 2, + }], + ])('still proves an empty round when %s', (_label, diagnostics) => { + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, diagnostics, { isPrivate: false })) + .toBe(true); + expect(catchupPlaneReady(emptyPeers, diagnostics, { isPrivate: false })).toBe(true); + }); + + it('is still proven when the curator DID answer, unreachable strangers aside', () => { + // The positive complement: the flag is about the curator's silence, not + // about the round being lossy. + const curatorAnswered = { ...cleanEmptyRound, failedPeers: 3, authorityUnanswered: false }; + expect(catchupPlaneProvenByUnanimousEmpty(emptyPeers, curatorAnswered, { isPrivate: false })) + .toBe(true); + }); + }); + it('accepts either evidence carrier for the clean empty completion', () => { // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter // (`diagnostics.emptyResponses`) are separate carriers, and the legacy From 01e19e4d558d04cc94090414a86435de12443b30 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sat, 1 Aug 2026 22:32:23 +0200 Subject: [PATCH 37/44] fix(sync): an ambiguous registry match may rank the walk but not end it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review round 25, and a trust escalation I introduced rather than a pre-existing bug. `resolveCuratorSyncPeer` resolves a wallet-address curator DID through `discovery.findAgents()` when the deterministic `DKG_CREATOR` route finds nothing. The code has always documented that pick as arbitrary when several agents register the same wallet — which was harmless while this function only RANKED the walk. It is not harmless now that `'metadata'` provenance means "may end the walk": an ordinary member sharing the curator's wallet could answer with a clean subset, be treated as the graph's authority, and stop the walk before the real curator was ever contacted. A third provenance, `'registry'`, now marks that case. It still ranks the walk — an arbitrary co-registrant is a better first try than nothing — but `authoritativeSyncPeerId` accepts only `'metadata'`, so it can never settle a plane. A SINGLE registration stays authoritative, because that is a deterministic binding, so the ordinary case keeps its early stop. Mutation, both directions: making the ambiguous match authoritative again kills the negative test, and demoting the unique match kills the positive one. A third test pins that neither deterministic route — a bare peer-id DID, or the projected `DKG_CREATOR` triple — touches the registry at all, so the guard cannot be widened into something that disables the early stop wholesale. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-cg-resolve.ts | 41 +++++++++++++++-- packages/agent/test/sync-policy.test.ts | 51 +++++++++++++++++++++- 2 files changed, 87 insertions(+), 5 deletions(-) diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index f670ee2b52..32070217d5 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -525,7 +525,15 @@ async function applyContextGraphListPrivacy( /** Where a resolved catch-up sync peer came from; see {@link resolveCuratorSyncPeer}. */ export interface SyncPeerResolution { peerId?: string; - provenance: 'metadata' | 'bootstrap-hint' | 'none'; + /** + * - `'metadata'` — resolved deterministically from `/_meta`. AUTHORITATIVE. + * - `'registry'` — a wallet-address curator matched more than one agent + * registration, so which peer we got is arbitrary. Ranks the + * walk; may NOT end it. + * - `'bootstrap-hint'` — the authenticated join-approval hint. Ranks only. + * - `'none'` — no peer at all. + */ + provenance: 'metadata' | 'registry' | 'bootstrap-hint' | 'none'; } /** @@ -537,6 +545,15 @@ export function authoritativeSyncPeerId(resolution: SyncPeerResolution): string return resolution.provenance === 'metadata' ? resolution.peerId : undefined; } +/** + * Does this DID identify a peer directly, or does it need resolving through a + * registry? Wallet-address curators (V10) are the indirect case; a bare libp2p + * peer id (legacy) is already the answer. + */ +function curatorDidNeedsRegistryResolution(curatorIdentifier: string): boolean { + return curatorIdentifier.startsWith('0x'); +} + /** * Resolve the curator peer for a Context Graph together with WHERE it came from. * @@ -544,6 +561,10 @@ export function authoritativeSyncPeerId(resolution: SyncPeerResolution): string * * - `'metadata'` — `/_meta` names a curator DID and it resolved to a peer. * Authoritative: that peer speaks for the whole graph. + * - `'registry'` — the curator DID named a WALLET address and more than one agent + * registration claimed it, so the peer we picked is arbitrary. Ranks the walk; + * never ends it. A single registration stays `'metadata'`, because that is a + * deterministic binding. * - `'bootstrap-hint'` — the authenticated join-approval hint recorded in * `preferredSyncPeers`, used while `_meta` has not arrived yet (and restored * from the durable join-approved membership row after restart). It is a fine @@ -588,7 +609,10 @@ export async function resolveCuratorSyncPeer( // stores the libp2p peer ID) over the agent registry (which may return // an arbitrary match when multiple agents register the same wallet). let curatorPeerId = curatorIdentifier; - if (curatorIdentifier.startsWith('0x')) { + // Deterministic until proven otherwise: a bare peer-id DID and the projected + // `DKG_CREATOR` route both come straight out of `/_meta`. + let provenance: SyncPeerResolution['provenance'] = 'metadata'; + if (curatorDidNeedsRegistryResolution(curatorIdentifier)) { let resolved = false; // Preferred: use the same projected metadata resolution as privacy and @@ -616,12 +640,21 @@ export async function resolveCuratorSyncPeer( throwIfSyncAuthAborted(options.signal); const agents = await agent.discovery.findAgents(); throwIfSyncAuthAborted(options.signal); - const match = agents.find( + const matches = agents.filter( (a) => a.agentAddress?.toLowerCase() === curatorIdentifier.toLowerCase(), ); + const match = matches[0]; if (match) { curatorPeerId = match.peerId; resolved = true; + // Which registration we get is arbitrary when a wallet has more than + // one. That was harmless while this only RANKED the walk; it is not + // harmless now that `'metadata'` means "may end the walk", because an + // ordinary member sharing the curator's wallet could answer with a + // clean subset and stop the walk before the real curator is reached. + // A single registration is still a deterministic binding, so the + // common case keeps the early stop. + if (matches.length > 1) provenance = 'registry'; } } catch { throwIfSyncAuthAborted(options.signal); @@ -633,7 +666,7 @@ export async function resolveCuratorSyncPeer( } bootstrapHints.delete(contextGraphId); - return { peerId: curatorPeerId, provenance: 'metadata' }; + return { peerId: curatorPeerId, provenance }; } export class ContextGraphResolveMethods extends DKGAgentBase { diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index f55d0fb696..b864d0e861 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -157,7 +157,9 @@ describe('curator sync-peer provenance', () => { .toEqual({ provenance: 'none' }); }); - it('resolves a wallet-address curator through the registry as authoritative', async () => { + it('resolves a UNIQUELY registered wallet curator as authoritative', async () => { + // One registration for the wallet is a deterministic binding, so the early + // stop is preserved for the ordinary case. const hints = new Map([[CG, HINT]]); const agent = agentWithMeta( { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, @@ -168,6 +170,53 @@ describe('curator sync-peer provenance', () => { .toEqual({ peerId: CURATOR, provenance: 'metadata' }); }); + it('will not make an AMBIGUOUS registry match an authority', async () => { + // `findAgents()` returns whichever registrations exist for a wallet, and the + // code has always documented the pick as arbitrary when there are several. + // That was harmless while this only ranked the walk. It is not harmless now + // that `'metadata'` means "may end the walk": an ordinary member sharing the + // curator's wallet could answer with a clean subset and stop the walk before + // the real curator is ever contacted. + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [ + { agentAddress: '0x00000000000000000000000000000000000000AB', peerId: '12D3KooWMemberA' }, + { agentAddress: '0x00000000000000000000000000000000000000ab', peerId: '12D3KooWMemberB' }, + ], + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + // It still RANKS the walk — an arbitrary co-registrant is a better first try + // than nothing… + expect(resolved.peerId).toBe('12D3KooWMemberA'); + expect(resolved.provenance).toBe('registry'); + // …but it can never END it. + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('keeps the deterministic metadata routes authoritative', async () => { + // The positive complement, so the guard cannot be widened into something + // that disables the early stop wholesale. Neither of these routes touches + // the registry: a bare peer-id DID, and the projected DKG_CREATOR triple. + const bareDid = agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }, async () => { + throw new Error('a bare peer-id DID must not need the registry'); + }); + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(bareDid as never, new Map(), CG), + )).toBe(CURATOR); + + const viaCreator = agentWithMeta({ + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, async () => { + throw new Error('the DKG_CREATOR route must not need the registry'); + }); + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(viaCreator as never, new Map(), CG), + )).toBe(CURATOR); + }); + it('answers ranking and authority from ONE resolution', async () => { // The catch-up boundary needs both notions, and resolving twice is not // free or even equivalent: each resolution reads `_meta` (and can drive the From 5c973485077a0750aa04780c242fd30897be711a Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 01:10:17 +0200 Subject: [PATCH 38/44] fix(sync): authority must come from the graph's OWN metadata, not the projection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merge-readiness review, P1 (blocking) and P2. P1. `provenance: 'metadata'` was supposed to mean a source-qualified curator binding. It did not. Two routes reached it without one: - `getCgMeta()` is a MERGED projection — it unions `/_meta` with the AGENTS, `_catalog` and ONTOLOGY graphs under first-wins precedence and discards which graph supplied each fact. A creator contributed by an AGENTS-only declaration was indistinguishable from one the Context Graph declared about itself. My own comment claimed both came "straight out of `/_meta`"; that was false. - The registry fallback demoted only when the LOCAL result had several matches, but `DiscoveryClient.findAgents()` queries a strictly local Agent Registry, so one local match is not evidence of a network-wide binding. Round 25 keyed on exactly the wrong cardinality. Both passed `authoritativeSyncPeerId`, so an ordinary same-wallet member or a stale projected creator could answer with a clean subset and end the walk before the real curator was reached. Authority is now EARNED: `getOwnMetaFacts` reads the Context Graph's own `_meta` graph alone, and `'metadata'` requires that graph to name the curator AND, for a wallet-address curator, to bind the creator peer itself. Everything else is `'projection'` or `'registry'` — ranks the walk, never ends it — and an unavailable or throwing read fails closed to ranking. This keeps the early stop for the V10 wallet-curator case the reviewer flagged as the cost of blanket demotion: a graph that declares its own binding still gets it. P2. The retry deadline could exceed its budget. `Date.now()` follows a backwards wall-clock step, `1e308` passed `Number.isInteger`, and a `NaN` `retry.maxWaitMs` through the exported seam made every delay `NaN`, which the default timer treats as "immediately" — a spin under persistent refusal. Now `performance.now()`, safe -integer validation on both the env and in-process paths, and invalid env input falls back to the documented default. Every guard mutation-checked. The in-process validation mutant is the clearest: without it, NaN/Infinity/unsafe budgets run until the 60 s test timeout. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../src/context-graph-meta-projection.ts | 46 +++++++ packages/agent/src/dkg-agent-cg-resolve.ts | 128 +++++++++++++++--- packages/agent/src/sync/catchup-policy.ts | 34 ++++- packages/agent/test/catchup-policy.test.ts | 53 ++++++++ .../agent/test/cg-resolve-refresh.test.ts | 16 ++- packages/agent/test/sync-policy.test.ts | 116 ++++++++++++++-- 6 files changed, 353 insertions(+), 40 deletions(-) diff --git a/packages/agent/src/context-graph-meta-projection.ts b/packages/agent/src/context-graph-meta-projection.ts index 412c12cef1..beb0352e2b 100644 --- a/packages/agent/src/context-graph-meta-projection.ts +++ b/packages/agent/src/context-graph-meta-projection.ts @@ -312,6 +312,52 @@ export class ContextGraphMetaProjection { return (await this.store.listGraphs(options)).filter((graphUri) => graphUri.startsWith(prefix)); } + /** + * Facts declared by the Context Graph's OWN `/_meta` graph, with nothing + * merged in. + * + * `get()` deliberately unions `_meta`, AGENTS, `_catalog` and ONTOLOGY under + * first-wins precedence, which is right for privacy and listing reads — an + * AGENTS-only declaration can legitimately mark a graph private. It is NOT + * right for deciding who speaks for the graph: the merged record discards + * WHICH graph supplied each fact, so a creator contributed by AGENTS or + * ONTOLOGY is indistinguishable from one the Context Graph declared about + * itself. + * + * Catch-up authority needs that distinction (issue #2006), so it reads here + * instead. Same loader, one source. + */ + async getOwnMetaFacts( + contextGraphId: string, + options: QueryOptions = {}, + ): Promise { + const uri = contextGraphDataUri(contextGraphId); + const metaGraph = contextGraphMetaGraphUri(contextGraphId); + assertSafeIri(uri); + assertSafeIri(metaGraph); + + const record: ContextGraphMetaRecord = { + id: contextGraphId, + uri, + declared: false, + isSystem: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), + creators: [], + curators: [], + allowedPeers: [], + allowedAgents: [], + participantAgents: [], + participantIdentityIds: [], + revokedAgents: [], + delegations: [], + subGraphs: [], + hasAgentGate: false, + hasPeerGate: false, + hasLegacyParticipantGate: false, + }; + await this.loadContextGraphFacts(metaGraph, uri, record, options); + return record; + } + private async rebuild(contextGraphId: string, options: QueryOptions): Promise { const uri = contextGraphDataUri(contextGraphId); const ontologyGraph = contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY); diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 32070217d5..fd04783dcf 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -526,14 +526,23 @@ async function applyContextGraphListPrivacy( export interface SyncPeerResolution { peerId?: string; /** - * - `'metadata'` — resolved deterministically from `/_meta`. AUTHORITATIVE. - * - `'registry'` — a wallet-address curator matched more than one agent - * registration, so which peer we got is arbitrary. Ranks the - * walk; may NOT end it. - * - `'bootstrap-hint'` — the authenticated join-approval hint. Ranks only. + * Only `'metadata'` is AUTHORITATIVE — see {@link authoritativeSyncPeerId}. + * Everything else ranks the walk and may never end it. + * + * - `'metadata'` — the Context Graph's OWN `/_meta` declares the + * curator→peer binding, and that binding is internally + * consistent. + * - `'projection'` — the binding came from the merged metadata projection, + * which unions `_meta` with AGENTS / `_catalog` / ONTOLOGY + * and discards which graph supplied each fact. Good enough + * to rank; not a statement the graph made about itself. + * - `'registry'` — a wallet-address curator resolved through the agent + * registry, which is queried STRICTLY LOCALLY, so even a + * single local match is not proof of a network-wide binding. + * - `'bootstrap-hint'` — the authenticated join-approval hint; can be stale. * - `'none'` — no peer at all. */ - provenance: 'metadata' | 'registry' | 'bootstrap-hint' | 'none'; + provenance: 'metadata' | 'projection' | 'registry' | 'bootstrap-hint' | 'none'; } /** @@ -561,10 +570,14 @@ function curatorDidNeedsRegistryResolution(curatorIdentifier: string): boolean { * * - `'metadata'` — `/_meta` names a curator DID and it resolved to a peer. * Authoritative: that peer speaks for the whole graph. - * - `'registry'` — the curator DID named a WALLET address and more than one agent - * registration claimed it, so the peer we picked is arbitrary. Ranks the walk; - * never ends it. A single registration stays `'metadata'`, because that is a - * deterministic binding. + * - `'projection'` / `'registry'` — a peer was resolved, but not from a source that + * can speak for the graph. `getCgMeta()` is a MERGED projection: it unions + * `/_meta` with the AGENTS, `_catalog` and ONTOLOGY graphs under first-wins + * precedence and discards which graph supplied each fact, so a creator + * contributed by an AGENTS-only declaration is indistinguishable from one the + * graph declared about itself. The agent registry is queried strictly locally, + * so even a unique local match is not evidence of a network-wide binding. + * Both rank the walk and neither may end it. * - `'bootstrap-hint'` — the authenticated join-approval hint recorded in * `preferredSyncPeers`, used while `_meta` has not arrived yet (and restored * from the durable join-approved membership row after restart). It is a fine @@ -578,6 +591,56 @@ function curatorDidNeedsRegistryResolution(curatorIdentifier: string): boolean { * "metadata confirmed the curator" from "metadata found nothing and the hint * was echoed back". Only the resolver knows which branch it took. */ + +/** + * Does the Context Graph's OWN `/_meta` graph declare this exact + * curator→peer binding? + * + * The merged projection cannot answer this: it unions `_meta` with AGENTS, + * `_catalog` and ONTOLOGY and drops the source of each fact, so a stale or + * third-party creator declaration is indistinguishable from the graph's own. + * Catch-up authority lets ONE peer stand for a whole graph, so it needs the + * stronger statement. + * + * Returns false on any read failure — fail closed to ranking, never to authority. + */ +async function ownMetaConfirmsCuratorBinding( + agent: DKGAgent, + contextGraphId: string, + curatorDid: string, + curatorPeerId: string, + options: { signal?: AbortSignal }, +): Promise { + // Fail closed if the receiver cannot answer: an agent (or a hand-built test + // receiver) without the source-qualified reader ranks, never authorises. + if (typeof agent.getOwnCgMetaFacts !== 'function') return false; + let own; + try { + own = await agent.getOwnCgMetaFacts(contextGraphId, { signal: options.signal }); + } catch { + throwIfSyncAuthAborted(options.signal); + return false; + } + + // The graph must name this curator itself… + const ownCurators = [own.curator, ...own.curators].filter(Boolean); + if (!ownCurators.includes(curatorDid)) return false; + + const didPrefix = 'did:dkg:agent:'; + const curatorIdentifier = curatorDid.slice(didPrefix.length); + // …and for a bare peer-id DID that IS the binding, with nothing to reconcile. + if (!curatorDidNeedsRegistryResolution(curatorIdentifier)) { + return curatorPeerId === curatorIdentifier; + } + + // For a wallet-address curator, the peer must come from a creator the graph + // declared about itself — not one contributed by AGENTS or ONTOLOGY. + return [own.creator, ...own.creators] + .filter((value): value is string => Boolean(value)) + .some((creatorDid) => creatorDid.startsWith(didPrefix) + && creatorDid.slice(didPrefix.length) === curatorPeerId); +} + export async function resolveCuratorSyncPeer( agent: DKGAgent, /** @@ -609,9 +672,10 @@ export async function resolveCuratorSyncPeer( // stores the libp2p peer ID) over the agent registry (which may return // an arbitrary match when multiple agents register the same wallet). let curatorPeerId = curatorIdentifier; - // Deterministic until proven otherwise: a bare peer-id DID and the projected - // `DKG_CREATOR` route both come straight out of `/_meta`. - let provenance: SyncPeerResolution['provenance'] = 'metadata'; + // Assume the weaker classification and EARN `'metadata'` below. The previous + // comment here claimed the projected `DKG_CREATOR` route came "straight out of + // `/_meta`"; it does not — `getCgMeta()` merges four graphs. + let provenance: SyncPeerResolution['provenance'] = 'projection'; if (curatorDidNeedsRegistryResolution(curatorIdentifier)) { let resolved = false; @@ -647,14 +711,11 @@ export async function resolveCuratorSyncPeer( if (match) { curatorPeerId = match.peerId; resolved = true; - // Which registration we get is arbitrary when a wallet has more than - // one. That was harmless while this only RANKED the walk; it is not - // harmless now that `'metadata'` means "may end the walk", because an - // ordinary member sharing the curator's wallet could answer with a - // clean subset and stop the walk before the real curator is reached. - // A single registration is still a deterministic binding, so the - // common case keeps the early stop. - if (matches.length > 1) provenance = 'registry'; + // NEVER authoritative, however many matches came back. `findAgents()` + // queries the LOCAL Agent Registry only, so "one match" means one match + // on this node — not that the wallet has a single registration on the + // network. Local cardinality cannot prove a binding. + provenance = 'registry'; } } catch { throwIfSyncAuthAborted(options.signal); @@ -665,6 +726,15 @@ export async function resolveCuratorSyncPeer( if (!resolved) return fromHint(); } + // Earn `'metadata'`: re-derive the binding from the Context Graph's OWN `_meta` + // graph and require it to agree. This is what makes the label mean "the graph + // said so", rather than "something in the merged projection said so". + if (provenance === 'projection') { + provenance = await ownMetaConfirmsCuratorBinding( + agent, contextGraphId, curatorDid, curatorPeerId, options, + ) ? 'metadata' : 'projection'; + } + bootstrapHints.delete(contextGraphId); return { peerId: curatorPeerId, provenance }; } @@ -678,6 +748,22 @@ export class ContextGraphResolveMethods extends DKGAgentBase { return this.contextGraphMetaProjection.get(contextGraphId, { signal: options.signal }); } + /** + * Facts from the Context Graph's OWN `/_meta` graph only — the + * source-qualified counterpart of {@link getCgMeta}, which merges four graphs. + * Used where a fact has to be attributable to the graph itself; see + * `resolveCuratorSyncPeer`. + */ + async getOwnCgMetaFacts( + this: DKGAgent, + contextGraphId: string, + options: { signal?: AbortSignal } = {}, + ): Promise { + return this.contextGraphMetaProjection.getOwnMetaFacts(contextGraphId, { + signal: options.signal, + }); + } + async listContextGraphsFromProjection(this: DKGAgent, opts?: { callerAgentAddress?: string | null }): Promise { // Before enabling this default-on: thread the caller signal into getCgMeta // and wrap per-row reads in withBudget (per A1's LIST_CONTEXT_GRAPHS_*_BUDGET_MS); diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index 7139ba5557..e63448cada 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -44,7 +44,10 @@ export function resolveCatchupBackpressureMaxWaitMs(raw: string | undefined): nu const trimmed = raw?.trim(); if (!trimmed) return DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; const parsed = Number(trimmed); - return Number.isInteger(parsed) && parsed >= 0 + // `Number.isInteger` alone accepts `1e308`, which is an integer by IEEE-754 and + // a budget no operator meant. Require a SAFE integer so an unusable value falls + // back to the documented default instead of becoming an unbounded wait. + return Number.isSafeInteger(parsed) && parsed >= 0 ? parsed : DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; } @@ -132,6 +135,17 @@ export function catchupSourceForMode(mode: CatchupMode): CatchupAdmissionSource return mode === 'foreground' ? 'catchup-foreground' : 'catchup-background'; } +/** + * Elapsed-time source for the retry deadline. + * + * `performance.now()` is monotonic; `Date.now()` is not, and the budget here is a + * duration rather than a point in time, so it must not follow a wall-clock + * correction. Falls back to `Date.now` only if `performance` is unavailable. + */ +const monotonicNow: () => number = typeof performance?.now === 'function' + ? () => performance.now() + : Date.now; + /** A pending backoff must never keep the process alive past `agent.stop()`. */ function defaultWait(delayMs: number): Promise { return new Promise((resolve) => { @@ -233,7 +247,23 @@ export async function runCatchupPlaneWithPolicy( ); } - const now = options.now ?? Date.now; + // `retry.maxWaitMs` reaches the loop without passing the env parser, so a + // `NaN` or unsafe value here would make every computed delay `NaN` — which the + // default timer treats as "as soon as possible", turning a bounded backoff into + // a spin under persistent refusal. + const configuredMaxWait = options.retry?.maxWaitMs; + if (configuredMaxWait !== undefined + && !(Number.isSafeInteger(configuredMaxWait) && configuredMaxWait >= 0)) { + throw new TypeError( + `runCatchupPlaneWithPolicy: retry.maxWaitMs must be a non-negative safe integer, got ${String(configuredMaxWait)}.`, + ); + } + + // Monotonic by default. `Date.now()` moves with the wall clock, so an NTP step + // BACKWARDS during a catch-up silently extends the advertised budget — the one + // direction a bound must not move. Tests still inject `now`, and the paired + // seam is enforced above. + const now = options.now ?? monotonicNow; const wait = options.wait ?? defaultWait; const maxWaitMs = options.retry?.maxWaitMs ?? CATCHUP_BACKPRESSURE_MAX_WAIT_MS; // Absolute deadline fixed once per plane, taken BEFORE the first admission diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index 13248f1ceb..5f6e998678 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -241,6 +241,59 @@ describe('runCatchupPlanesWithPolicy', () => { }); }); +describe('the retry budget is bounded even under bad input or a moving clock', () => { + it.each([ + ['an unsafe magnitude from the environment', '1e308'], + ['a fractional value', '12.5'], + ['a negative value', '-1'], + ['a non-number', 'soon'], + ])('falls back to the documented default for %s', (_label, raw) => { + // `Number.isInteger` alone accepts 1e308 — an integer by IEEE-754 and a + // budget nobody meant. An unusable env value must become the default, never + // an unbounded wait. + expect(resolveCatchupBackpressureMaxWaitMs(raw)) + .toBe(DEFAULT_CATCHUP_BACKPRESSURE_MAX_WAIT_MS); + }); + + it.each([ + ['NaN', Number.NaN], + ['Infinity', Number.POSITIVE_INFINITY], + ['an unsafe integer', 1e308], + ['a negative budget', -1], + ['a fractional budget', 5.5], + ])('rejects %s supplied through the in-process retry seam', async (_label, maxWaitMs) => { + // This path bypasses the env parser entirely. A NaN budget makes every + // computed delay NaN, which the default timer treats as "immediately" — + // turning a bounded backoff into a spin under persistent refusal. + await expect( + runCatchupPlaneWithPolicy('foreground', async () => ({ deferredBackpressure: 1 }), { + retry: { maxWaitMs: maxWaitMs as number }, + }), + ).rejects.toThrow(/non-negative safe integer/); + }); + + it('reads the deadline from a MONOTONIC clock, not the wall clock', async () => { + // The budget is a DURATION, so it must not follow an NTP correction: a + // backwards wall-clock step would silently hand back the time it rewound, + // extending the advertised budget in the one direction a bound must not move. + // + // Asserted by what production READS rather than by simulating a rollback — + // an injected `now` is honoured verbatim, so injection would bypass the very + // protection under test. + const dateNow = vi.spyOn(Date, 'now'); + try { + await runCatchupPlaneWithPolicy( + 'foreground', + async () => ({ deferredBackpressure: 1 }), + { retry: { maxWaitMs: 0 } }, + ); + expect(dateNow).not.toHaveBeenCalled(); + } finally { + dateNow.mockRestore(); + } + }); +}); + describe('the wait/now clock seam', () => { // Before #2006 `retryDelaysMs` bounded the loop, so an injected `wait` that // resolved instantly still terminated after three steps. The ladder is gone and diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index ac56b67a15..ba130c991f 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1231,14 +1231,18 @@ describe('refreshMetaFromCurator', () => { const bootstrapPeer = 'peer-from-join-approval'; const authoritativePeer = 'peer-from-authoritative-meta'; const preferredSyncPeers = new Map([[contextGraphId, bootstrapPeer]]); + const declaredFacts = { + curator: 'did:dkg:agent:0x0000000000000000000000000000000000000abc', + curators: [], + creator: `did:dkg:agent:${authoritativePeer}`, + creators: [], + }; const agent = { preferredSyncPeers, - getCgMeta: async () => ({ - curator: 'did:dkg:agent:0x0000000000000000000000000000000000000abc', - curators: [], - creator: `did:dkg:agent:${authoritativePeer}`, - creators: [], - }), + getCgMeta: async () => declaredFacts, + // The Context Graph declares this curator→peer binding in its OWN `_meta`, + // which is what makes it authoritative rather than merely rankable (#2006). + getOwnCgMetaFacts: async () => declaredFacts, discovery: { findAgents: async () => { throw new Error('creator metadata should resolve the curator peer'); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index b864d0e861..347d667b11 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -103,14 +103,27 @@ describe('curator sync-peer provenance', () => { const HINT = '12D3KooWBootstrapHint'; const CURATOR = '12D3KooWMetadataCurator'; - function agentWithMeta(meta: { + type MetaFacts = { curator?: string; curators?: string[]; creator?: string; creators?: string[]; - }, findAgents: () => Promise> = async () => []) { + }; + + /** + * `getCgMeta` is the MERGED projection (`_meta` + AGENTS + `_catalog` + + * ONTOLOGY); `getOwnCgMetaFacts` is what the Context Graph declared about + * itself. `ownMeta` defaults to `meta` — the ordinary case where they agree — + * so any test exercising the difference has to say so out loud. + */ + function agentWithMeta( + meta: MetaFacts, + findAgents: () => Promise> = async () => [], + ownMeta: MetaFacts = meta, + ) { return { getCgMeta: async () => ({ curators: [], creators: [], ...meta }), + getOwnCgMetaFacts: async () => ({ curators: [], creators: [], ...ownMeta }), discovery: { findAgents }, }; } @@ -157,17 +170,94 @@ describe('curator sync-peer provenance', () => { .toEqual({ provenance: 'none' }); }); - it('resolves a UNIQUELY registered wallet curator as authoritative', async () => { - // One registration for the wallet is a deterministic binding, so the early - // stop is preserved for the ordinary case. + it('never lets a registry match be an authority, however unique it looks locally', async () => { + // `findAgents()` queries the LOCAL Agent Registry only, so "one match" means + // one match on THIS node — not that the wallet has a single registration on + // the network. Local cardinality cannot establish a binding, so this route + // ranks and never settles. const hints = new Map([[CG, HINT]]); const agent = agentWithMeta( { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, async () => [{ agentAddress: '0x00000000000000000000000000000000000000AB', peerId: CURATOR }], ); - expect(await resolveCuratorSyncPeer(agent as never, hints, CG)) - .toEqual({ peerId: CURATOR, provenance: 'metadata' }); + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('registry'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('authorises a wallet curator the graph binds to a peer in its OWN _meta', async () => { + // The route that keeps the early stop for V10 wallet-address curators: the + // Context Graph itself declares both the curator DID and the creator peer. + const hints = new Map([[CG, HINT]]); + const declared = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const agent = agentWithMeta(declared, async () => { + throw new Error('an own-_meta binding must not need the registry'); + }, declared); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'metadata' }); + expect(authoritativeSyncPeerId(resolved)).toBe(CURATOR); + }); + + it('demotes a creator the MERGED projection supplied but the graph did not', async () => { + // `getCgMeta()` unions `_meta` with AGENTS / `_catalog` / ONTOLOGY under + // first-wins precedence and discards which graph supplied each fact. A + // creator contributed by an AGENTS-only declaration therefore looks identical + // to one the graph declared about itself — but only the latter may end the + // walk. Here the projection offers a creator the graph's own `_meta` does not. + const hints = new Map([[CG, HINT]]); + const agent = agentWithMeta( + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => [], + // The graph names the curator, but binds no creator peer itself. + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('demotes when the graph does not name that curator at all', async () => { + // A curator the merged projection asserts but the graph never claimed. + const agent = agentWithMeta( + { curator: `did:dkg:agent:${CURATOR}` }, + async () => [], + {}, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('fails closed to ranking when the source-qualified read is unavailable', async () => { + // A receiver without the reader, or one whose read throws, must never be + // upgraded to authority. + const noReader = { + getCgMeta: async () => ({ curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }), + discovery: { findAgents: async () => [] }, + }; + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(noReader as never, new Map(), CG), + )).toBeUndefined(); + + const throwingReader = { + ...noReader, + getOwnCgMetaFacts: async () => { throw new Error('store unavailable'); }, + }; + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(throwingReader as never, new Map(), CG), + )).toBeUndefined(); }); it('will not make an AMBIGUOUS registry match an authority', async () => { @@ -206,12 +296,13 @@ describe('curator sync-peer provenance', () => { await resolveCuratorSyncPeer(bareDid as never, new Map(), CG), )).toBe(CURATOR); - const viaCreator = agentWithMeta({ + const declared = { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', creator: `did:dkg:agent:${CURATOR}`, - }, async () => { + }; + const viaCreator = agentWithMeta(declared, async () => { throw new Error('the DKG_CREATOR route must not need the registry'); - }); + }, declared); expect(authoritativeSyncPeerId( await resolveCuratorSyncPeer(viaCreator as never, new Map(), CG), )).toBe(CURATOR); @@ -224,12 +315,15 @@ describe('curator sync-peer provenance', () => { // metadata confirms a curator, so the second call runs against a different // map than the first. let metaReads = 0; + const declared = { curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }; const agent = { preferredSyncPeers: new Map([[CG, CURATOR]]), getCgMeta: async () => { metaReads += 1; - return { curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }; + return declared; }, + // The graph declares this curator itself, so the binding is authoritative. + getOwnCgMetaFacts: async () => declared, discovery: { findAgents: async () => [] }, }; From 6be6d1923333a92013bf9f4cf48a9726cd8360c2 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 01:52:29 +0200 Subject: [PATCH 39/44] fix(sync): read the graph's real definition, and stop freezing empty verdicts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from the latest review round, two of them corrections to fixes landed in this PR. Public graphs never became authoritative (regression from 5c9734850). A Context Graph writes its definition to ONE graph, chosen by access policy — `defGraph = isCurated ? cgMetaGraph : ontologyGraph`. The source-qualified reader added last round read `/_meta` alone, so it found nothing for the ordinary PUBLIC layout and demoted every such graph to ranking. That loses the early stop AND the per-plane narrowing, which would re-open the amplification this PR exists to fix. The reader now covers both definition graphs and still excludes the third-party carriers (AGENTS, `_catalog`). Because ONTOLOGY is network-replicated, authority additionally requires the declared creator to be UNIQUE: an injected creator then shows up alongside the real one and demotes to ranking instead of being resolved by picking whichever happens to match. Same discipline as the conflicting-creator guard in `context-graph-public-meta-repair.ts`. Raw empty counters could bypass per-peer completion evidence. `emptyResponses` counts an empty payload; `emptyPeers` counts a peer whose round was empty AND clean. ORing them let an explicitly incomplete empty response prove a plane ready. The aggregate is now consulted only when there is no completion evidence at all, and the legacy branch passes no completion rather than an all-zero stand-in that would have suppressed its own fallback. Empty-derived readiness is no longer permanent. Readiness provenance is carried forward by an OR against the previous run, so a unanimous-empty verdict — derived from ABSENCE of evidence, and reachable with one unrelated empty response plus transport-level peer failures when no authority resolves — became permanent for the subscription. Such a verdict now settles the current run but is not persisted, so it is re-derived each time and a wrong verdict heals on the next catch-up. Readiness proven by content or by the curator's own hosted-empty word stays sticky. Failing closed on unaccounted peers without an authority was rejected as the alternative: it makes the unanimous-empty mode unusable on any lossy network, leaving those graphs permanently `unreachable` and retried forever — the resource drain this issue set out to reduce. Also extracts the duplicated `ContextGraphMetaRecord` initializer, and covers the definition reader against a real store rather than a stub — a stub cannot get the source of a fact wrong, which is what let the `_meta`-only version look correct. All four guards mutation-checked; the conflict row initially SURVIVED because its fixture took the registry short-circuit and never reached the code under test. Refs #2006 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../src/context-graph-meta-projection.ts | 103 ++++++++++-------- packages/agent/src/dkg-agent-cg-resolve.ts | 43 +++++--- .../agent/test/cg-resolve-refresh.test.ts | 2 +- .../context-graph-meta-projection.test.ts | 93 ++++++++++++++++ packages/agent/test/sync-policy.test.ts | 61 ++++++++++- packages/cli/src/catchup-runner.ts | 13 ++- packages/cli/src/context-graph-readiness.ts | 96 ++++++++++++---- packages/cli/test/catchup-runner.test.ts | 42 ++++++- .../context-graph-catchup-readiness.test.ts | 72 +++++++++++- 9 files changed, 431 insertions(+), 94 deletions(-) diff --git a/packages/agent/src/context-graph-meta-projection.ts b/packages/agent/src/context-graph-meta-projection.ts index beb0352e2b..c5a9a7f6ab 100644 --- a/packages/agent/src/context-graph-meta-projection.ts +++ b/packages/agent/src/context-graph-meta-projection.ts @@ -128,6 +128,37 @@ const CATALOG_META_PREDICATES = new Set([ DKG_ONTOLOGY.DCT_ACCESS_RIGHTS, ]); +/** + * A `ContextGraphMetaRecord` with no facts loaded yet. + * + * Shared by every reader so a field added to the record cannot be initialized + * in one loader and forgotten in another. + */ +function emptyContextGraphMetaRecord( + contextGraphId: string, + uri: string, +): ContextGraphMetaRecord { + const isSystem = (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId); + return { + id: contextGraphId, + uri, + declared: isSystem, + isSystem, + creators: [], + curators: [], + allowedPeers: [], + allowedAgents: [], + participantAgents: [], + participantIdentityIds: [], + revokedAgents: [], + delegations: [], + subGraphs: [], + hasAgentGate: false, + hasPeerGate: false, + hasLegacyParticipantGate: false, + }; +} + export class ContextGraphMetaProjection { private readonly entries = new Map(); @@ -313,48 +344,51 @@ export class ContextGraphMetaProjection { } /** - * Facts declared by the Context Graph's OWN `/_meta` graph, with nothing - * merged in. + * Facts the Context Graph declared about ITSELF, read from its definition + * graphs alone. * * `get()` deliberately unions `_meta`, AGENTS, `_catalog` and ONTOLOGY under * first-wins precedence, which is right for privacy and listing reads — an * AGENTS-only declaration can legitimately mark a graph private. It is NOT * right for deciding who speaks for the graph: the merged record discards * WHICH graph supplied each fact, so a creator contributed by AGENTS or - * ONTOLOGY is indistinguishable from one the Context Graph declared about - * itself. + * `_catalog` (both of which carry THIRD-PARTY assertions — other agents' + * self-declarations and peer-fetchable catalog records) is indistinguishable + * from one the Context Graph declared about itself. + * + * A Context Graph's definition is written to exactly one graph, chosen by + * access policy (`dkg-agent-context-graph.ts`): + * + * const defGraph = isCurated ? cgMetaGraph : ontologyGraph; + * + * so a CURATED graph declares itself in `/_meta` and a PUBLIC one in + * ONTOLOGY. Reading only `_meta` would therefore find nothing for the + * ordinary public case. Both are read here, subject-scoped to this graph's + * URI; AGENTS and `_catalog` stay excluded. * - * Catch-up authority needs that distinction (issue #2006), so it reads here - * instead. Same loader, one source. + * Because ONTOLOGY is network-replicated (a public graph's definition is + * broadcast by its creator), the two sources can disagree. Callers deciding + * authority must therefore require a UNIQUE creator across the union rather + * than accepting any match — see `ownMetaConfirmsCuratorBinding`. That + * mirrors the conflict discipline already applied in + * `context-graph-public-meta-repair.ts`. + * + * Catch-up authority needs this distinction (issue #2006). */ - async getOwnMetaFacts( + async getOwnDefinitionFacts( contextGraphId: string, options: QueryOptions = {}, ): Promise { const uri = contextGraphDataUri(contextGraphId); const metaGraph = contextGraphMetaGraphUri(contextGraphId); + const ontologyGraph = contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY); assertSafeIri(uri); assertSafeIri(metaGraph); + assertSafeIri(ontologyGraph); - const record: ContextGraphMetaRecord = { - id: contextGraphId, - uri, - declared: false, - isSystem: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), - creators: [], - curators: [], - allowedPeers: [], - allowedAgents: [], - participantAgents: [], - participantIdentityIds: [], - revokedAgents: [], - delegations: [], - subGraphs: [], - hasAgentGate: false, - hasPeerGate: false, - hasLegacyParticipantGate: false, - }; + const record = emptyContextGraphMetaRecord(contextGraphId, uri); await this.loadContextGraphFacts(metaGraph, uri, record, options); + await this.loadContextGraphFacts(ontologyGraph, uri, record, options); return record; } @@ -376,24 +410,7 @@ export class ContextGraphMetaProjection { assertSafeIri(metaGraph); assertSafeIri(catalogGraph); - const record: ContextGraphMetaRecord = { - id: contextGraphId, - uri, - declared: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), - isSystem: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId), - creators: [], - curators: [], - allowedPeers: [], - allowedAgents: [], - participantAgents: [], - participantIdentityIds: [], - revokedAgents: [], - delegations: [], - subGraphs: [], - hasAgentGate: false, - hasPeerGate: false, - hasLegacyParticipantGate: false, - }; + const record = emptyContextGraphMetaRecord(contextGraphId, uri); // Authoritative (local, fully trusted) sources first, meta-first so its // scalars win via first-wins (`??=`) precedence. The floor-filtered `_catalog` diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index fd04783dcf..fc3bee1c52 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -613,10 +613,10 @@ async function ownMetaConfirmsCuratorBinding( ): Promise { // Fail closed if the receiver cannot answer: an agent (or a hand-built test // receiver) without the source-qualified reader ranks, never authorises. - if (typeof agent.getOwnCgMetaFacts !== 'function') return false; + if (typeof agent.getOwnCgDefinitionFacts !== 'function') return false; let own; try { - own = await agent.getOwnCgMetaFacts(contextGraphId, { signal: options.signal }); + own = await agent.getOwnCgDefinitionFacts(contextGraphId, { signal: options.signal }); } catch { throwIfSyncAuthAborted(options.signal); return false; @@ -633,12 +633,27 @@ async function ownMetaConfirmsCuratorBinding( return curatorPeerId === curatorIdentifier; } - // For a wallet-address curator, the peer must come from a creator the graph - // declared about itself — not one contributed by AGENTS or ONTOLOGY. - return [own.creator, ...own.creators] - .filter((value): value is string => Boolean(value)) - .some((creatorDid) => creatorDid.startsWith(didPrefix) - && creatorDid.slice(didPrefix.length) === curatorPeerId); + // For a wallet-address curator the peer must come from a creator the GRAPH + // declared, and that declaration must be unambiguous. + // + // A public graph's definition lives in ONTOLOGY, which is network-replicated: + // its creator broadcasts it, so any node can assert a `DKG_CREATOR` for this + // subject. A curated graph's definition lives in `/_meta` and is never + // broadcast. `getOwnCgDefinitionFacts` reads both, so an injected ONTOLOGY + // creator shows up ALONGSIDE the real one rather than instead of it — and a + // second, conflicting creator is exactly what must not be resolved by + // picking whichever one happens to match. Requiring a unique creator turns + // that attack into a demotion to ranking, which costs fan-out and never + // costs correctness. Same discipline as the conflicting-creator guard in + // `context-graph-public-meta-repair.ts`. + const declaredCreators = new Set( + [own.creator, ...own.creators].filter((value): value is string => Boolean(value)), + ); + if (declaredCreators.size !== 1) return false; + + const [declaredCreator] = [...declaredCreators]; + return declaredCreator.startsWith(didPrefix) + && declaredCreator.slice(didPrefix.length) === curatorPeerId; } export async function resolveCuratorSyncPeer( @@ -749,17 +764,17 @@ export class ContextGraphResolveMethods extends DKGAgentBase { } /** - * Facts from the Context Graph's OWN `/_meta` graph only — the - * source-qualified counterpart of {@link getCgMeta}, which merges four graphs. - * Used where a fact has to be attributable to the graph itself; see - * `resolveCuratorSyncPeer`. + * Facts the Context Graph declared about ITSELF — the source-qualified + * counterpart of {@link getCgMeta}, which merges four graphs and discards + * which one supplied each fact. Used where a fact has to be attributable to + * the graph itself; see `resolveCuratorSyncPeer`. */ - async getOwnCgMetaFacts( + async getOwnCgDefinitionFacts( this: DKGAgent, contextGraphId: string, options: { signal?: AbortSignal } = {}, ): Promise { - return this.contextGraphMetaProjection.getOwnMetaFacts(contextGraphId, { + return this.contextGraphMetaProjection.getOwnDefinitionFacts(contextGraphId, { signal: options.signal, }); } diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index ba130c991f..9d7f06cc33 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1242,7 +1242,7 @@ describe('refreshMetaFromCurator', () => { getCgMeta: async () => declaredFacts, // The Context Graph declares this curator→peer binding in its OWN `_meta`, // which is what makes it authoritative rather than merely rankable (#2006). - getOwnCgMetaFacts: async () => declaredFacts, + getOwnCgDefinitionFacts: async () => declaredFacts, discovery: { findAgents: async () => { throw new Error('creator metadata should resolve the curator peer'); diff --git a/packages/agent/test/context-graph-meta-projection.test.ts b/packages/agent/test/context-graph-meta-projection.test.ts index 46ab39e3c9..dcda879fca 100644 --- a/packages/agent/test/context-graph-meta-projection.test.ts +++ b/packages/agent/test/context-graph-meta-projection.test.ts @@ -534,3 +534,96 @@ describe('ContextGraphMetaProjection', () => { expect((await projection.get(id)).accessPolicy).toBe('private'); }); }); + +describe('getOwnDefinitionFacts', () => { + const CURATOR_DID = 'did:dkg:agent:0x00000000000000000000000000000000000000ab'; + const CREATOR_DID = 'did:dkg:agent:12D3KooWCuratorPeer'; + + /** + * A Context Graph's definition is written to ONE graph, chosen by access + * policy (`dkg-agent-context-graph.ts`): + * + * const defGraph = isCurated ? cgMetaGraph : ontologyGraph; + * + * These cover the reader against a real store rather than through a stubbed + * agent, because the whole point of the reader is WHICH graph a fact came + * from — a stub cannot get that wrong, and a stub is what let an earlier + * `_meta`-only version look correct while missing every public graph. + */ + it('reads a PUBLIC graph definition, which lives in ONTOLOGY', async () => { + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = 'own-definition-public'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + ]); + + const own = await projection.getOwnDefinitionFacts(id); + expect(own.curators).toEqual([CURATOR_DID]); + expect(own.creators).toEqual([CREATOR_DID]); + }); + + it('reads a CURATED graph definition, which lives in the graph\'s own _meta', async () => { + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = '0x00000000000000000000000000000000000000ab/own-definition-curated'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphMetaGraphUri(id) }, + ]); + + const own = await projection.getOwnDefinitionFacts(id); + expect(own.curators).toEqual([CURATOR_DID]); + expect(own.creators).toEqual([CREATOR_DID]); + }); + + it('ignores creators contributed by AGENTS or the peer-fetchable _catalog', async () => { + // The reason this reader exists: `get()` merges these in and discards which + // graph supplied each fact, so a third-party assertion becomes + // indistinguishable from the graph's own declaration. + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = 'own-definition-third-party'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.RDF_TYPE, object: DKG_ONTOLOGY.DKG_CONTEXT_GRAPH, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWAgentsClaim', graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.AGENTS) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWCatalogClaim', graph: contextGraphCatalogUri(id) }, + ]); + + expect((await projection.getOwnDefinitionFacts(id)).creators).toEqual([]); + // …while the merged projection does surface the AGENTS claim, which is the + // precise difference the authority decision turns on. + expect((await projection.get(id)).creators).toContain('did:dkg:agent:12D3KooWAgentsClaim'); + }); + + it('surfaces a conflicting ONTOLOGY creator rather than hiding it', async () => { + // ONTOLOGY is network-replicated, so any node can assert a creator for a + // subject. The reader must not silently prefer one: it returns BOTH so the + // authority decision can refuse an ambiguous binding. + const store = new OxigraphStore(); + const projection = new ContextGraphMetaProjection(store); + const id = '0x00000000000000000000000000000000000000ab/own-definition-conflict'; + const subject = contextGraphDataUri(id); + + await store.insert([ + { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphMetaGraphUri(id) }, + { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWInjectedPeer', graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, + ]); + + const own = await projection.getOwnDefinitionFacts(id); + expect(own.creators).toHaveLength(2); + expect(own.creators).toContain(CREATOR_DID); + expect(own.creators).toContain('did:dkg:agent:12D3KooWInjectedPeer'); + }); +}); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 347d667b11..24ad5681cc 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -112,7 +112,7 @@ describe('curator sync-peer provenance', () => { /** * `getCgMeta` is the MERGED projection (`_meta` + AGENTS + `_catalog` + - * ONTOLOGY); `getOwnCgMetaFacts` is what the Context Graph declared about + * ONTOLOGY); `getOwnCgDefinitionFacts` is what the Context Graph declared about * itself. `ownMeta` defaults to `meta` — the ordinary case where they agree — * so any test exercising the difference has to say so out loud. */ @@ -123,7 +123,7 @@ describe('curator sync-peer provenance', () => { ) { return { getCgMeta: async () => ({ curators: [], creators: [], ...meta }), - getOwnCgMetaFacts: async () => ({ curators: [], creators: [], ...ownMeta }), + getOwnCgDefinitionFacts: async () => ({ curators: [], creators: [], ...ownMeta }), discovery: { findAgents }, }; } @@ -227,6 +227,59 @@ describe('curator sync-peer provenance', () => { expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); + it('authorises a PUBLIC graph whose creator binding lives in ONTOLOGY', async () => { + // A public Context Graph writes its definition to ONTOLOGY, not to + // `/_meta` (`defGraph = isCurated ? cgMetaGraph : ontologyGraph`), so a + // reader scoped to `_meta` alone finds no creator and every ordinary public + // graph loses its authority — and with it the early stop AND the per-plane + // narrowing this issue exists to gain. `getOwnCgDefinitionFacts` reads both + // definition graphs; here the curator comes from `_meta` and the creator + // from ONTOLOGY, which is the ordinary public layout. + const declared = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const agent = agentWithMeta(declared, async () => { + throw new Error('a declared binding must not need the registry'); + }, declared); + + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(agent as never, new Map(), CG), + )).toBe(CURATOR); + }); + + it('refuses to pick a side when the declared creator is ambiguous', async () => { + // ONTOLOGY is network-replicated, so a hostile node can assert a second + // `DKG_CREATOR` for a graph that already declares one in `_meta`. Resolving + // that by accepting whichever creator happens to match the candidate peer + // would let the injected fact authorise the attacker. Two creators means no + // binding: rank, never settle. + const agent = agentWithMeta( + // The MERGED projection resolves the peer, so the walk still reaches the + // own-declaration check rather than short-circuiting into the registry + // fallback — which demotes unconditionally and would make this pass for + // the wrong reason. + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => { + throw new Error('the registry fallback must not be reached here'); + }, + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWInjectedPeer'], + }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + // The candidate peer DOES appear among the declared creators; accepting it + // on that basis is exactly the mutation this row exists to kill. + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + it('demotes when the graph does not name that curator at all', async () => { // A curator the merged projection asserts but the graph never claimed. const agent = agentWithMeta( @@ -253,7 +306,7 @@ describe('curator sync-peer provenance', () => { const throwingReader = { ...noReader, - getOwnCgMetaFacts: async () => { throw new Error('store unavailable'); }, + getOwnCgDefinitionFacts: async () => { throw new Error('store unavailable'); }, }; expect(authoritativeSyncPeerId( await resolveCuratorSyncPeer(throwingReader as never, new Map(), CG), @@ -323,7 +376,7 @@ describe('curator sync-peer provenance', () => { return declared; }, // The graph declares this curator itself, so the binding is authoritative. - getOwnCgMetaFacts: async () => declared, + getOwnCgDefinitionFacts: async () => declared, discovery: { findAgents: async () => [] }, }; diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index 1087e21671..e2eec4019d 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -781,8 +781,17 @@ export function catchupPlaneProvenByUnanimousEmpty( // A non-curator that has `_meta` and no data cannot tell "the graph is empty" // from "I have not synced it yet". See the note above. if ((diagnostics?.metaOnlyResponses ?? 0) > 0) return false; - const cleanEmptyObserved = (completion?.emptyPeers ?? 0) > 0 - || (diagnostics?.emptyResponses ?? 0) > 0; + // Completion evidence is PER-PEER and says explicitly whether that peer's + // round was clean; `diagnostics.emptyResponses` is a raw aggregate that counts + // an empty payload even when the peer's round was NOT complete. Where the + // runner supplied completion evidence it is the whole truth for this plane, so + // the aggregate must not re-admit a response the per-peer view already + // excluded — otherwise an explicitly incomplete empty result proves the plane + // ready. The aggregate is a fallback for legacy callers that carry no + // completion evidence at all, never a second chance for callers that do. + const cleanEmptyObserved = completion !== undefined + ? (completion.emptyPeers ?? 0) > 0 + : (diagnostics?.emptyResponses ?? 0) > 0; if (!cleanEmptyObserved) return false; // The one peer whose silence is decisive. See the note above for why this is // scoped to the curator rather than to `failedPeers`. diff --git a/packages/cli/src/context-graph-readiness.ts b/packages/cli/src/context-graph-readiness.ts index 2c4362b42f..c4672320bf 100644 --- a/packages/cli/src/context-graph-readiness.ts +++ b/packages/cli/src/context-graph-readiness.ts @@ -6,6 +6,9 @@ import type { } from '@origintrail-official/dkg-node-ui'; import { catchupPlaneCompletedWithoutFailure, + catchupPlaneProvenByAuthorityHostedEmpty, + catchupPlaneProvenByData, + catchupPlaneProvenByUnanimousEmpty, catchupPlaneReady, type CatchupJobResult, type CatchupPlaneCompletionEvidence, @@ -184,15 +187,50 @@ export function catchupResultHasCleanResponse(result: CatchupJobResult): boolean (!result.denied && peerReturnedMetadata); } -function catchupPlaneReadyThisRun(input: { +interface CatchupPlaneReadinessThisRun { + /** Whether this plane counts as ready for THIS run's reported job status. */ + ready: boolean; + /** + * Whether the evidence is strong enough to PERSIST as sticky readiness + * provenance. + * + * Readiness provenance is carried forward by an OR against + * `readinessBeforeCatchup`, so anything recorded here is permanent for the + * subscription. Only positive evidence earns that: verified content, or the + * curator's own word that it hosts an empty graph. + * + * A unanimous-empty round does NOT. It is a verdict derived from ABSENCE of + * evidence over the peers that answered, and with no authoritative curator to + * anchor it a single unrelated empty response alongside transport-level peer + * failures can produce it. That is tolerable as a per-run verdict — it lets a + * genuinely empty public graph settle instead of retrying forever — but it + * must be re-derived every run rather than frozen, so a wrong verdict heals on + * the next catch-up instead of becoming permanent (issue #2006). + * + * The alternative — failing closed on unaccounted peers whenever no authority + * resolves — was rejected: it makes the unanimous-empty mode unusable on any + * lossy network, leaving such graphs permanently `unreachable` and retried + * forever, which is the resource drain this issue set out to reduce. + */ + persistable: boolean; +} + +function catchupPlaneReadinessThisRun(input: { result: CatchupJobResult; plane: 'durable' | 'sharedMemory'; isPrivate: boolean; -}): boolean { +}): CatchupPlaneReadinessThisRun { const diagnostics = input.result.diagnostics?.[input.plane]; const completion = input.result.cleanPlaneCompletions?.[input.plane]; + const options = { isPrivate: input.isPrivate }; if (completion) { - return catchupPlaneReady(completion, diagnostics, { isPrivate: input.isPrivate }); + const provenPositively = catchupPlaneProvenByData(completion) + || catchupPlaneProvenByAuthorityHostedEmpty(completion, diagnostics, options); + return { + ready: provenPositively + || catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options), + persistable: provenPositively, + }; } // Backward compatibility for callers that construct a legacy result (for @@ -205,12 +243,19 @@ function catchupPlaneReadyThisRun(input: { ? input.result.dataSynced > 0 || (input.result.diagnostics?.durable.verifiedPrivateOnlyResponses ?? 0) > 0 : input.result.sharedMemorySynced > 0; - if (catchupPlaneCompletedWithoutFailure(diagnostics) && dataProgress) return true; - return catchupPlaneReady( - { verifiedDataPeers: 0, verifiedPrivateOnlyPeers: 0, emptyPeers: 0 }, - diagnostics, - { isPrivate: input.isPrivate }, - ); + if (catchupPlaneCompletedWithoutFailure(diagnostics) && dataProgress) { + return { ready: true, persistable: true }; + } + // Pass NO completion evidence rather than an all-zero stand-in: the empty + // proof consults the raw aggregate counters only when completion evidence is + // genuinely absent, and a synthetic `emptyPeers: 0` would read as "the + // per-peer view saw no clean empty response" and suppress the legacy path. + return { + ready: catchupPlaneReady(undefined, diagnostics, options), + // No completion evidence means neither positive proof mode can fire, so + // there is nothing here that may be frozen into provenance. + persistable: false, + }; } export interface ContextGraphCatchupReadinessClassification { @@ -267,25 +312,34 @@ export function classifyContextGraphCatchupReadiness(input: { }; } - const durableReadyThisRun = catchupPlaneReadyThisRun({ + const durableThisRun = catchupPlaneReadinessThisRun({ result, plane: 'durable', isPrivate: input.isPrivate, }); - const sharedMemoryReadyThisRun = input.includeSharedMemory && - catchupPlaneReadyThisRun({ + const sharedMemoryThisRun = input.includeSharedMemory + ? catchupPlaneReadinessThisRun({ result, plane: 'sharedMemory', isPrivate: input.isPrivate, - }); + }) + : { ready: false, persistable: false }; + const durableReadyThisRun = durableThisRun.ready; + const sharedMemoryReadyThisRun = sharedMemoryThisRun.ready; const currentReadinessProvenance = input.readinessBeforeCatchup.version >= CONTEXT_GRAPH_READINESS_VERSION; - const durableVerified = - (currentReadinessProvenance && input.readinessBeforeCatchup.durableVerified) || - durableReadyThisRun; - const sharedMemoryVerified = - (currentReadinessProvenance && input.readinessBeforeCatchup.sharedMemoryVerified) || - sharedMemoryReadyThisRun; + const durableVerifiedBefore = + currentReadinessProvenance && input.readinessBeforeCatchup.durableVerified; + const sharedMemoryVerifiedBefore = + currentReadinessProvenance && input.readinessBeforeCatchup.sharedMemoryVerified; + const durableVerified = durableVerifiedBefore || durableReadyThisRun; + const sharedMemoryVerified = sharedMemoryVerifiedBefore || sharedMemoryReadyThisRun; + // What this run is allowed to FREEZE, as opposed to what it reports. These + // diverge only for a unanimous-empty verdict, which stays re-derived per run + // so that a wrong empty verdict cannot become permanent. + const durableVerifiedPersisted = durableVerifiedBefore || durableThisRun.persistable; + const sharedMemoryVerifiedPersisted = + sharedMemoryVerifiedBefore || sharedMemoryThisRun.persistable; const overallVerified = durableVerified || sharedMemoryVerified; const missingGraphProof = !overallVerified; const missingRequestedSharedMemory = @@ -319,8 +373,8 @@ export function classifyContextGraphCatchupReadiness(input: { pendingMeta: false, }, readinessPatch: { - durableVerified, - sharedMemoryVerified, + durableVerified: durableVerifiedPersisted, + sharedMemoryVerified: sharedMemoryVerifiedPersisted, }, eventPayload: durableReadyThisRun || sharedMemoryReadyThisRun ? { diff --git a/packages/cli/test/catchup-runner.test.ts b/packages/cli/test/catchup-runner.test.ts index 1ce98c800a..e4eacd6eff 100644 --- a/packages/cli/test/catchup-runner.test.ts +++ b/packages/cli/test/catchup-runner.test.ts @@ -957,23 +957,55 @@ describe('catch-up plane proof predicates', () => { }); }); - it('accepts either evidence carrier for the clean empty completion', () => { + it('uses per-peer completion evidence, and the aggregate ONLY without it', () => { // Per-peer evidence (`cleanPlaneCompletions`) and the aggregate counter - // (`diagnostics.emptyResponses`) are separate carriers, and the legacy - // no-`cleanPlaneCompletions` branch in the readiness classifier can only - // supply the aggregate one. Pin each independently so neither disjunct can - // be dropped unnoticed. + // (`diagnostics.emptyResponses`) are separate carriers, but they are not + // interchangeable and must not be ORed together. + // + // `emptyResponses` counts an empty PAYLOAD; `emptyPeers` counts a peer whose + // round was empty AND clean. A peer that answered empty but did not complete + // raises the first and not the second — so consulting the aggregate when + // per-peer evidence exists lets an explicitly incomplete response prove the + // plane ready, which is the false-`done` class this proof exists to prevent. expect(catchupPlaneProvenByUnanimousEmpty( { ...noEvidence, emptyPeers: 1 }, { ...cleanEmptyRound, emptyResponses: 0 }, { isPrivate: false }, )).toBe(true); + + // Completion evidence PRESENT and negative: the aggregate must not re-admit + // it. This is the assertion that fails if the carriers are ORed. expect(catchupPlaneProvenByUnanimousEmpty( noEvidence, { ...cleanEmptyRound, emptyResponses: 1 }, { isPrivate: false }, + )).toBe(false); + + // Completion evidence genuinely ABSENT (the legacy runner result): the + // aggregate is the only carrier there is, so it still counts. Without this + // row, dropping the fallback entirely would look like a passing change. + expect(catchupPlaneProvenByUnanimousEmpty( + undefined, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, )).toBe(true); }); + + it('does not let an explicitly incomplete empty peer prove the plane', () => { + // The production shape of the row above: the worker reports a peer that + // returned an empty payload but whose round never completed, so the peer is + // absent from `emptyPeers` while `emptyResponses` still counts it. + const incompleteEmpty = catchupPeerPlaneEvidence( + { emptyResponses: 1, completedPhases: 0, bytesReceived: 0 }, + { plane: 'durable', complete: false }, + ); + expect(incompleteEmpty.emptyPeers).toBe(0); + expect(catchupPlaneProvenByUnanimousEmpty( + incompleteEmpty, + { ...cleanEmptyRound, emptyResponses: 1 }, + { isPrivate: false }, + )).toBe(false); + }); }); describe('catch-up peer accounting with a skipped plane', () => { diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 0fc8fbb49f..3a7fd42004 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -1,6 +1,9 @@ import { describe, expect, it } from 'vitest'; import type { CatchupJobResult } from '../src/catchup-runner.js'; -import { classifyContextGraphCatchupReadiness } from '../src/context-graph-readiness.js'; +import { + CONTEXT_GRAPH_READINESS_VERSION, + classifyContextGraphCatchupReadiness, +} from '../src/context-graph-readiness.js'; function mixedPeerResult(verifiedDataPeers: number): CatchupJobResult { return { @@ -209,7 +212,7 @@ describe('context graph catch-up readiness classification', () => { return result; } - it('accepts a unanimously clean-empty public round as proof the plane is empty', () => { + it('settles a unanimously clean-empty public round WITHOUT freezing it', () => { const classification = classifyContextGraphCatchupReadiness({ result: publicEmptyRoundResult(), includeSharedMemory: false, @@ -224,13 +227,73 @@ describe('context graph catch-up readiness classification', () => { synced: true, sharedMemorySynced: false, }, + // Reported ready, but NOT persisted as provenance: readiness derived from + // the absence of evidence has to be re-derived every run. See the + // two-run regression below. readinessPatch: { - durableVerified: true, + durableVerified: false, sharedMemoryVerified: false, }, }); }); + it('re-derives an empty verdict instead of carrying it into the next run', () => { + // The false-`done` residual: with no authoritative curator, one unrelated + // empty response alongside transport-level peer failures satisfies the + // unanimous-empty proof. That is survivable as a per-run verdict, but the + // readiness OR against `readinessBeforeCatchup` would otherwise make it + // permanent — every later run would report `done` without re-proving + // anything, which is exactly the false-`done` class issue #2006 targets. + const firstRun = publicEmptyRoundResult(); + firstRun.cleanPlaneCompletions!.durable.emptyPeers = 1; + firstRun.diagnostics!.durable.emptyResponses = 1; + firstRun.diagnostics!.durable.failedPeers = 1; + + const first = classifyContextGraphCatchupReadiness({ + result: firstRun, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + expect(first.jobStatus).toBe('done'); + expect(first.readinessPatch).toMatchObject({ durableVerified: false }); + + // Second run: metadata only, nothing proven. Feed back exactly what run one + // persisted. If the empty verdict had been frozen, this would still say + // `done` while proving nothing. + const secondRun = publicEmptyRoundResult(); + secondRun.cleanPlaneCompletions!.durable.emptyPeers = 0; + secondRun.diagnostics!.durable.emptyResponses = 0; + secondRun.diagnostics!.durable.metaOnlyResponses = 1; + + const second = classifyContextGraphCatchupReadiness({ + result: secondRun, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup: { + ...readinessBeforeCatchup, + version: CONTEXT_GRAPH_READINESS_VERSION, + durableVerified: first.readinessPatch!.durableVerified!, + }, + }); + expect(second.jobStatus).not.toBe('done'); + + // The other half of the contract: readiness proven by CONTENT is still + // sticky, so a graph that really did sync does not re-prove itself forever. + const proven = publicEmptyRoundResult(); + proven.dataSynced = 12; + proven.cleanPlaneCompletions!.durable.verifiedDataPeers = 1; + expect(classifyContextGraphCatchupReadiness({ + result: proven, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }).readinessPatch).toMatchObject({ durableVerified: true }); + }); + it('does not accept a public clean-empty peer when another peer denies', () => { // A denial means we did not hear from every peer, so "nobody has anything" // is not established. Before #2006 this returned `done`. @@ -305,7 +368,8 @@ describe('context graph catch-up readiness classification', () => { readinessBeforeCatchup, })).toMatchObject({ jobStatus: 'done', - readinessPatch: { durableVerified: true }, + // Same rule as the non-legacy path: settled for this run, not frozen. + readinessPatch: { durableVerified: false }, }); }); From 461ec648ace5c1cf386b57ba77d417e5ad7bbf14 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 02:21:56 +0200 Subject: [PATCH 40/44] fix(sync): keep catch-up authority out of network-replicated metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the ONTOLOGY half of 6be6d1923. That commit widened the source-qualified reader to `_meta` ∪ ONTOLOGY so PUBLIC graphs — which write their definition to ONTOLOGY — could earn authority, and required a UNIQUE declared creator to contain the risk. Uniqueness does not contain it. ONTOLOGY is network-replicated, so a node can hold an injected `DKG_CREATOR` for a subject WITHOUT ever holding the real one; the injected row is then the only one visible and uniqueness is satisfied by the attacker. "The only creator I can currently see" is a statement about local cardinality, which is precisely the reasoning that already makes the Agent Registry route non-authoritative — reintroduced one graph over. The reader is `_meta`-only again, and the creator check is back to "declared by the graph itself". The cost is real and is stated rather than engineered around: a public graph whose identity facts live only in replicated ONTOLOGY earns no authority, so its catch-up degrades to the previous bounded fan-out. That is today's behaviour rather than a regression — the fan-out reduction is earned by graphs that declare their own binding — but it does mean the amplification fix does not engage for that shape. An attempt to recover the reduction without trust — letting the resolved peer's verified DATA narrow the walk even when unauthoritative, on the grounds that the readiness classifier already settles on any peer's complete verified payload — was implemented and then withdrawn. It is wrong: settling the VERDICT and stopping the WALK are different things, and stopping early on an untrusted peer accepts one member's snapshot as the whole graph. An existing regression test ("does not let a bootstrap-hint preferred peer stop the walk") caught it, which is exactly what it was written for. Only a curator's identity ever justified that stop. Mutation-checked: re-adding the ONTOLOGY source to the reader fails both new projection tests. Refs #2006 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../src/context-graph-meta-projection.ts | 37 ++++++------- packages/agent/src/dkg-agent-cg-resolve.ts | 53 +++++++++--------- .../agent/test/cg-resolve-refresh.test.ts | 2 +- .../context-graph-meta-projection.test.ts | 43 +++++++++------ packages/agent/test/sync-policy.test.ts | 55 +++++-------------- 5 files changed, 83 insertions(+), 107 deletions(-) diff --git a/packages/agent/src/context-graph-meta-projection.ts b/packages/agent/src/context-graph-meta-projection.ts index c5a9a7f6ab..d7ab1a0b12 100644 --- a/packages/agent/src/context-graph-meta-projection.ts +++ b/packages/agent/src/context-graph-meta-projection.ts @@ -344,8 +344,7 @@ export class ContextGraphMetaProjection { } /** - * Facts the Context Graph declared about ITSELF, read from its definition - * graphs alone. + * Facts the Context Graph declared about ITSELF in its own `_meta` graph. * * `get()` deliberately unions `_meta`, AGENTS, `_catalog` and ONTOLOGY under * first-wins precedence, which is right for privacy and listing reads — an @@ -356,39 +355,35 @@ export class ContextGraphMetaProjection { * self-declarations and peer-fetchable catalog records) is indistinguishable * from one the Context Graph declared about itself. * - * A Context Graph's definition is written to exactly one graph, chosen by - * access policy (`dkg-agent-context-graph.ts`): + * Only `/_meta` is read. ONTOLOGY is deliberately NOT included even + * though a PUBLIC graph writes its definition there + * (`defGraph = isCurated ? cgMetaGraph : ontologyGraph`): ONTOLOGY is + * network-replicated, so any node can assert a `DKG_CREATOR` for a subject, + * and a row being the only one currently visible LOCALLY proves nothing + * about what the network holds. Requiring local uniqueness there would + * repeat, one graph over, the same local-cardinality fallacy that makes the + * Agent Registry route non-authoritative. * - * const defGraph = isCurated ? cgMetaGraph : ontologyGraph; - * - * so a CURATED graph declares itself in `/_meta` and a PUBLIC one in - * ONTOLOGY. Reading only `_meta` would therefore find nothing for the - * ordinary public case. Both are read here, subject-scoped to this graph's - * URI; AGENTS and `_catalog` stay excluded. - * - * Because ONTOLOGY is network-replicated (a public graph's definition is - * broadcast by its creator), the two sources can disagree. Callers deciding - * authority must therefore require a UNIQUE creator across the union rather - * than accepting any match — see `ownMetaConfirmsCuratorBinding`. That - * mirrors the conflict discipline already applied in - * `context-graph-public-meta-repair.ts`. + * The consequence is deliberate, and is a real cost: a public graph whose + * identity facts live only in replicated ONTOLOGY has NO locally trustworthy + * binding, so it earns no authority and its catch-up degrades to the previous + * bounded fan-out. That is today's behaviour rather than a regression — the + * fan-out reduction is earned by graphs that declare their own binding, and + * settling a graph on an unverifiable claim is the worse trade. * * Catch-up authority needs this distinction (issue #2006). */ - async getOwnDefinitionFacts( + async getOwnMetaFacts( contextGraphId: string, options: QueryOptions = {}, ): Promise { const uri = contextGraphDataUri(contextGraphId); const metaGraph = contextGraphMetaGraphUri(contextGraphId); - const ontologyGraph = contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY); assertSafeIri(uri); assertSafeIri(metaGraph); - assertSafeIri(ontologyGraph); const record = emptyContextGraphMetaRecord(contextGraphId, uri); await this.loadContextGraphFacts(metaGraph, uri, record, options); - await this.loadContextGraphFacts(ontologyGraph, uri, record, options); return record; } diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index fc3bee1c52..926744fc60 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -613,10 +613,10 @@ async function ownMetaConfirmsCuratorBinding( ): Promise { // Fail closed if the receiver cannot answer: an agent (or a hand-built test // receiver) without the source-qualified reader ranks, never authorises. - if (typeof agent.getOwnCgDefinitionFacts !== 'function') return false; + if (typeof agent.getOwnCgMetaFacts !== 'function') return false; let own; try { - own = await agent.getOwnCgDefinitionFacts(contextGraphId, { signal: options.signal }); + own = await agent.getOwnCgMetaFacts(contextGraphId, { signal: options.signal }); } catch { throwIfSyncAuthAborted(options.signal); return false; @@ -633,27 +633,26 @@ async function ownMetaConfirmsCuratorBinding( return curatorPeerId === curatorIdentifier; } - // For a wallet-address curator the peer must come from a creator the GRAPH - // declared, and that declaration must be unambiguous. + // For a wallet-address curator, the peer must come from a creator the graph + // declared in its OWN `_meta` — not one contributed by AGENTS, `_catalog`, or + // ONTOLOGY. // - // A public graph's definition lives in ONTOLOGY, which is network-replicated: - // its creator broadcasts it, so any node can assert a `DKG_CREATOR` for this - // subject. A curated graph's definition lives in `/_meta` and is never - // broadcast. `getOwnCgDefinitionFacts` reads both, so an injected ONTOLOGY - // creator shows up ALONGSIDE the real one rather than instead of it — and a - // second, conflicting creator is exactly what must not be resolved by - // picking whichever one happens to match. Requiring a unique creator turns - // that attack into a demotion to ranking, which costs fan-out and never - // costs correctness. Same discipline as the conflicting-creator guard in - // `context-graph-public-meta-repair.ts`. - const declaredCreators = new Set( - [own.creator, ...own.creators].filter((value): value is string => Boolean(value)), - ); - if (declaredCreators.size !== 1) return false; - - const [declaredCreator] = [...declaredCreators]; - return declaredCreator.startsWith(didPrefix) - && declaredCreator.slice(didPrefix.length) === curatorPeerId; + // ONTOLOGY is excluded even though a PUBLIC graph's definition lives there, + // and local uniqueness does not rescue it: ONTOLOGY is network-replicated, so + // a node can hold an injected `DKG_CREATOR` for a subject WITHOUT holding the + // real one, and "the only creator I can currently see" would then authorise + // the attacker. That is the same local-cardinality fallacy that already makes + // the Agent Registry route non-authoritative, one graph over. + // + // So a public graph with a wallet curator earns no authority here, and its + // catch-up degrades to the previous bounded fan-out. The resolved peer still + // ORDERS the walk, but ordering is all it may do: ending the walk means + // accepting one peer's snapshot as the whole graph, and an untrusted peer's + // `complete` flag says only that it served its own view. + return [own.creator, ...own.creators] + .filter((value): value is string => Boolean(value)) + .some((creatorDid) => creatorDid.startsWith(didPrefix) + && creatorDid.slice(didPrefix.length) === curatorPeerId); } export async function resolveCuratorSyncPeer( @@ -764,17 +763,17 @@ export class ContextGraphResolveMethods extends DKGAgentBase { } /** - * Facts the Context Graph declared about ITSELF — the source-qualified - * counterpart of {@link getCgMeta}, which merges four graphs and discards - * which one supplied each fact. Used where a fact has to be attributable to + * Facts from the Context Graph's OWN `/_meta` graph only — the + * source-qualified counterpart of {@link getCgMeta}, which merges four + * graphs and discards which one supplied each fact. Used where a fact has to be attributable to * the graph itself; see `resolveCuratorSyncPeer`. */ - async getOwnCgDefinitionFacts( + async getOwnCgMetaFacts( this: DKGAgent, contextGraphId: string, options: { signal?: AbortSignal } = {}, ): Promise { - return this.contextGraphMetaProjection.getOwnDefinitionFacts(contextGraphId, { + return this.contextGraphMetaProjection.getOwnMetaFacts(contextGraphId, { signal: options.signal, }); } diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index 9d7f06cc33..ba130c991f 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1242,7 +1242,7 @@ describe('refreshMetaFromCurator', () => { getCgMeta: async () => declaredFacts, // The Context Graph declares this curator→peer binding in its OWN `_meta`, // which is what makes it authoritative rather than merely rankable (#2006). - getOwnCgDefinitionFacts: async () => declaredFacts, + getOwnCgMetaFacts: async () => declaredFacts, discovery: { findAgents: async () => { throw new Error('creator metadata should resolve the curator peer'); diff --git a/packages/agent/test/context-graph-meta-projection.test.ts b/packages/agent/test/context-graph-meta-projection.test.ts index dcda879fca..aadd3faef4 100644 --- a/packages/agent/test/context-graph-meta-projection.test.ts +++ b/packages/agent/test/context-graph-meta-projection.test.ts @@ -535,7 +535,7 @@ describe('ContextGraphMetaProjection', () => { }); }); -describe('getOwnDefinitionFacts', () => { +describe('getOwnMetaFacts', () => { const CURATOR_DID = 'did:dkg:agent:0x00000000000000000000000000000000000000ab'; const CREATOR_DID = 'did:dkg:agent:12D3KooWCuratorPeer'; @@ -550,7 +550,13 @@ describe('getOwnDefinitionFacts', () => { * from — a stub cannot get that wrong, and a stub is what let an earlier * `_meta`-only version look correct while missing every public graph. */ - it('reads a PUBLIC graph definition, which lives in ONTOLOGY', async () => { + it('does NOT read ONTOLOGY, even though a public graph defines itself there', async () => { + // A public Context Graph writes its definition to ONTOLOGY, so it is + // tempting to read it here. ONTOLOGY is network-replicated, though: this + // node can hold an injected `DKG_CREATOR` for a subject WITHOUT holding the + // real one, and then "the only creator I can see" is the attacker's. Local + // cardinality proves nothing about the network — the same reason the Agent + // Registry route is non-authoritative. const store = new OxigraphStore(); const projection = new ContextGraphMetaProjection(store); const id = 'own-definition-public'; @@ -562,9 +568,12 @@ describe('getOwnDefinitionFacts', () => { { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, ]); - const own = await projection.getOwnDefinitionFacts(id); - expect(own.curators).toEqual([CURATOR_DID]); - expect(own.creators).toEqual([CREATOR_DID]); + const own = await projection.getOwnMetaFacts(id); + expect(own.curators).toEqual([]); + expect(own.creators).toEqual([]); + // The merged projection still sees them — that is the difference the + // authority decision turns on, and the reason this reader exists. + expect((await projection.get(id)).creators).toEqual([CREATOR_DID]); }); it('reads a CURATED graph definition, which lives in the graph\'s own _meta', async () => { @@ -579,7 +588,7 @@ describe('getOwnDefinitionFacts', () => { { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphMetaGraphUri(id) }, ]); - const own = await projection.getOwnDefinitionFacts(id); + const own = await projection.getOwnMetaFacts(id); expect(own.curators).toEqual([CURATOR_DID]); expect(own.creators).toEqual([CREATOR_DID]); }); @@ -600,30 +609,30 @@ describe('getOwnDefinitionFacts', () => { { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWCatalogClaim', graph: contextGraphCatalogUri(id) }, ]); - expect((await projection.getOwnDefinitionFacts(id)).creators).toEqual([]); + expect((await projection.getOwnMetaFacts(id)).creators).toEqual([]); // …while the merged projection does surface the AGENTS claim, which is the // precise difference the authority decision turns on. expect((await projection.get(id)).creators).toContain('did:dkg:agent:12D3KooWAgentsClaim'); }); - it('surfaces a conflicting ONTOLOGY creator rather than hiding it', async () => { - // ONTOLOGY is network-replicated, so any node can assert a creator for a - // subject. The reader must not silently prefer one: it returns BOTH so the - // authority decision can refuse an ambiguous binding. + it('ignores an injected ONTOLOGY creator even when it is the ONLY one visible', async () => { + // The attack this reader exists to stop: the graph's own `_meta` has not + // synced (or names only the curator), and the sole `DKG_CREATOR` this node + // can see for the subject was asserted by someone else. A reader that took + // ONTOLOGY would hand that peer the authority to settle the whole graph on + // an empty answer. const store = new OxigraphStore(); const projection = new ContextGraphMetaProjection(store); - const id = '0x00000000000000000000000000000000000000ab/own-definition-conflict'; + const id = '0x00000000000000000000000000000000000000ab/own-definition-injected'; const subject = contextGraphDataUri(id); await store.insert([ { subject, predicate: DKG_ONTOLOGY.DKG_CURATOR, object: CURATOR_DID, graph: contextGraphMetaGraphUri(id) }, - { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: CREATOR_DID, graph: contextGraphMetaGraphUri(id) }, { subject, predicate: DKG_ONTOLOGY.DKG_CREATOR, object: 'did:dkg:agent:12D3KooWInjectedPeer', graph: contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY) }, ]); - const own = await projection.getOwnDefinitionFacts(id); - expect(own.creators).toHaveLength(2); - expect(own.creators).toContain(CREATOR_DID); - expect(own.creators).toContain('did:dkg:agent:12D3KooWInjectedPeer'); + const own = await projection.getOwnMetaFacts(id); + expect(own.curators).toEqual([CURATOR_DID]); + expect(own.creators).toEqual([]); }); }); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 24ad5681cc..2454e52ace 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -112,7 +112,7 @@ describe('curator sync-peer provenance', () => { /** * `getCgMeta` is the MERGED projection (`_meta` + AGENTS + `_catalog` + - * ONTOLOGY); `getOwnCgDefinitionFacts` is what the Context Graph declared about + * ONTOLOGY); `getOwnCgMetaFacts` is what the Context Graph declared about * itself. `ownMeta` defaults to `meta` — the ordinary case where they agree — * so any test exercising the difference has to say so out loud. */ @@ -123,7 +123,7 @@ describe('curator sync-peer provenance', () => { ) { return { getCgMeta: async () => ({ curators: [], creators: [], ...meta }), - getOwnCgDefinitionFacts: async () => ({ curators: [], creators: [], ...ownMeta }), + getOwnCgMetaFacts: async () => ({ curators: [], creators: [], ...ownMeta }), discovery: { findAgents }, }; } @@ -227,38 +227,15 @@ describe('curator sync-peer provenance', () => { expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); - it('authorises a PUBLIC graph whose creator binding lives in ONTOLOGY', async () => { - // A public Context Graph writes its definition to ONTOLOGY, not to - // `/_meta` (`defGraph = isCurated ? cgMetaGraph : ontologyGraph`), so a - // reader scoped to `_meta` alone finds no creator and every ordinary public - // graph loses its authority — and with it the early stop AND the per-plane - // narrowing this issue exists to gain. `getOwnCgDefinitionFacts` reads both - // definition graphs; here the curator comes from `_meta` and the creator - // from ONTOLOGY, which is the ordinary public layout. - const declared = { - curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', - creator: `did:dkg:agent:${CURATOR}`, - }; - const agent = agentWithMeta(declared, async () => { - throw new Error('a declared binding must not need the registry'); - }, declared); - - expect(authoritativeSyncPeerId( - await resolveCuratorSyncPeer(agent as never, new Map(), CG), - )).toBe(CURATOR); - }); - - it('refuses to pick a side when the declared creator is ambiguous', async () => { - // ONTOLOGY is network-replicated, so a hostile node can assert a second - // `DKG_CREATOR` for a graph that already declares one in `_meta`. Resolving - // that by accepting whichever creator happens to match the candidate peer - // would let the injected fact authorise the attacker. Two creators means no - // binding: rank, never settle. + it('ranks but never settles a graph whose creator is not in its own _meta', async () => { + // The public shape: the merged projection resolves a creator peer (from the + // network-replicated ONTOLOGY graph), but the graph's own `_meta` declares + // no creator. That peer is still the best one to walk FIRST — it is + // returned as the resolved peer — but it may not end the walk on a claim + // that the graph is empty. The fan-out reduction for this shape comes from + // the peer's verified DATA narrowing the rest, which needs no trust; see + // `preferredEvidence` in `catchup-runner-worker-impl.ts`. const agent = agentWithMeta( - // The MERGED projection resolves the peer, so the walk still reaches the - // own-declaration check rather than short-circuiting into the registry - // fallback — which demotes unconditionally and would make this pass for - // the wrong reason. { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', creator: `did:dkg:agent:${CURATOR}`, @@ -266,16 +243,12 @@ describe('curator sync-peer provenance', () => { async () => { throw new Error('the registry fallback must not be reached here'); }, - { - curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', - creators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWInjectedPeer'], - }, + // The graph itself names the curator but binds no creator peer. + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, ); const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); expect(resolved.peerId).toBe(CURATOR); - // The candidate peer DOES appear among the declared creators; accepting it - // on that basis is exactly the mutation this row exists to kill. expect(resolved.provenance).toBe('projection'); expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); @@ -306,7 +279,7 @@ describe('curator sync-peer provenance', () => { const throwingReader = { ...noReader, - getOwnCgDefinitionFacts: async () => { throw new Error('store unavailable'); }, + getOwnCgMetaFacts: async () => { throw new Error('store unavailable'); }, }; expect(authoritativeSyncPeerId( await resolveCuratorSyncPeer(throwingReader as never, new Map(), CG), @@ -376,7 +349,7 @@ describe('curator sync-peer provenance', () => { return declared; }, // The graph declares this curator itself, so the binding is authoritative. - getOwnCgDefinitionFacts: async () => declared, + getOwnCgMetaFacts: async () => declared, discovery: { findAgents: async () => [] }, }; From cae12118116efc03ee0b0dd99af17d3b029a26c2 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 02:34:56 +0200 Subject: [PATCH 41/44] fix(sync): stop a partial empty round from marking the subscription synced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `subscription.synced` is a SECOND persisted readiness bit, living outside the provenance store. `contextGraphRowIsWritable` treats `subscribed && synced` as writable (`http-utils.ts:669`), and the legacy provenance migration reconstructs `durableVerified` from it (`context-graph-readiness.ts:641/663`). Making `readinessPatch` strict while leaving `statePatch.synced` on the transient verdict therefore left the withheld bit reachable by both routes. It also broke an invariant the code already relies on: `classifyExistingContextGraphReadiness` derives `synced` FROM the persisted provenance and patches the row back into agreement, so the two diverging would be corrected away on the next pass — after a window in which the graph looked writable. `synced` now carries the same verdict as `readinessPatch`. Basing `synced` on persistable proof alone would have left a genuinely empty graph permanently unsynced and unwritable, so the empty verdict earns persistence when the round was FULLY ACCOUNTED — every attempted peer answered (`failedPeers === 0`). Emptiness is only as good as the denominator it was taken over. That is the reviewer's earlier "fail closed on unaccounted peers" option, applied to PERSISTENCE rather than to the verdict, which keeps both properties that were pulling against each other: - liveness — the per-run verdict is unchanged, so a graph on a lossy network still reports `done` instead of retrying forever; - no frozen guess — nothing derived from a partial round is written down, by either route. Mutation-checked: dropping the fully-accounted requirement, and putting `statePatch.synced` back on the transient verdict, each fail the regressions. Refs #2006 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/cli/src/context-graph-readiness.ts | 62 ++++++++++++------- .../context-graph-catchup-readiness.test.ts | 38 ++++++++++-- 2 files changed, 73 insertions(+), 27 deletions(-) diff --git a/packages/cli/src/context-graph-readiness.ts b/packages/cli/src/context-graph-readiness.ts index c4672320bf..91aab03a66 100644 --- a/packages/cli/src/context-graph-readiness.ts +++ b/packages/cli/src/context-graph-readiness.ts @@ -196,21 +196,27 @@ interface CatchupPlaneReadinessThisRun { * * Readiness provenance is carried forward by an OR against * `readinessBeforeCatchup`, so anything recorded here is permanent for the - * subscription. Only positive evidence earns that: verified content, or the - * curator's own word that it hosts an empty graph. + * subscription. Verified content earns it outright, as does the curator's own + * word that it hosts an empty graph. * - * A unanimous-empty round does NOT. It is a verdict derived from ABSENCE of - * evidence over the peers that answered, and with no authoritative curator to - * anchor it a single unrelated empty response alongside transport-level peer - * failures can produce it. That is tolerable as a per-run verdict — it lets a - * genuinely empty public graph settle instead of retrying forever — but it - * must be re-derived every run rather than frozen, so a wrong verdict heals on - * the next catch-up instead of becoming permanent (issue #2006). + * A unanimous-empty round earns it only when the round was FULLY ACCOUNTED: + * every peer the walk attempted actually answered (`failedPeers === 0`). + * Emptiness is a verdict derived from ABSENCE of evidence, so it is only as + * good as the denominator it was taken over — with peers unaccounted for and + * no authoritative curator to anchor it, a single unrelated empty response + * produces the same verdict as a genuinely empty graph. * - * The alternative — failing closed on unaccounted peers whenever no authority - * resolves — was rejected: it makes the unanimous-empty mode unusable on any - * lossy network, leaving such graphs permanently `unreachable` and retried - * forever, which is the resource drain this issue set out to reduce. + * Splitting it this way keeps both properties that pulled against each other: + * + * - LIVENESS. The per-run verdict is unchanged, so a graph on a lossy network + * still reports `done` instead of retrying forever. Failing the verdict + * itself closed on unaccounted peers was rejected for exactly that reason. + * - NO FROZEN GUESS. Nothing derived from a partial round is written down, so + * a wrong empty verdict cannot outlive the run that produced it. + * + * This bit is what `statePatch.synced` is built from, and `synced` gates + * write preflight (`contextGraphRowIsWritable`), so anything admitted here + * grants durable readiness to consumers that never see the job result. */ persistable: boolean; } @@ -223,13 +229,16 @@ function catchupPlaneReadinessThisRun(input: { const diagnostics = input.result.diagnostics?.[input.plane]; const completion = input.result.cleanPlaneCompletions?.[input.plane]; const options = { isPrivate: input.isPrivate }; + // Every attempted peer answered, so the empty verdict was taken over the + // whole peer set rather than over whoever happened to reply. + const fullyAccounted = (diagnostics?.failedPeers ?? 0) === 0; if (completion) { const provenPositively = catchupPlaneProvenByData(completion) || catchupPlaneProvenByAuthorityHostedEmpty(completion, diagnostics, options); + const unanimousEmpty = catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options); return { - ready: provenPositively - || catchupPlaneProvenByUnanimousEmpty(completion, diagnostics, options), - persistable: provenPositively, + ready: provenPositively || unanimousEmpty, + persistable: provenPositively || (unanimousEmpty && fullyAccounted), }; } @@ -250,11 +259,13 @@ function catchupPlaneReadinessThisRun(input: { // proof consults the raw aggregate counters only when completion evidence is // genuinely absent, and a synthetic `emptyPeers: 0` would read as "the // per-peer view saw no clean empty response" and suppress the legacy path. + const ready = catchupPlaneReady(undefined, diagnostics, options); return { - ready: catchupPlaneReady(undefined, diagnostics, options), + ready, // No completion evidence means neither positive proof mode can fire, so - // there is nothing here that may be frozen into provenance. - persistable: false, + // anything true here came from the aggregate empty counter and is subject + // to the same fully-accounted requirement. + persistable: ready && fullyAccounted, }; } @@ -340,6 +351,15 @@ export function classifyContextGraphCatchupReadiness(input: { const durableVerifiedPersisted = durableVerifiedBefore || durableThisRun.persistable; const sharedMemoryVerifiedPersisted = sharedMemoryVerifiedBefore || sharedMemoryThisRun.persistable; + // `subscription.synced` is a SECOND persisted readiness bit, living outside + // the provenance store and consumed by callers that never see this job's + // result — `contextGraphRowIsWritable` treats `subscribed && synced` as + // writable. It must therefore carry the same verdict as `readinessPatch`, + // not the transient one. The pre-catch-up path already assumes they agree: + // it derives `synced` from the persisted provenance and patches the row + // back into line, so letting them diverge here would be corrected away on + // the next pass anyway — after a window in which the graph looked writable. + const overallVerifiedPersisted = durableVerifiedPersisted || sharedMemoryVerifiedPersisted; const overallVerified = durableVerified || sharedMemoryVerified; const missingGraphProof = !overallVerified; const missingRequestedSharedMemory = @@ -367,8 +387,8 @@ export function classifyContextGraphCatchupReadiness(input: { jobStatus, error, statePatch: { - synced: overallVerified, - sharedMemorySynced: sharedMemoryVerified, + synced: overallVerifiedPersisted, + sharedMemorySynced: sharedMemoryVerifiedPersisted, metaSynced: true, pendingMeta: false, }, diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 3a7fd42004..0c8df00287 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -212,7 +212,7 @@ describe('context graph catch-up readiness classification', () => { return result; } - it('settles a unanimously clean-empty public round WITHOUT freezing it', () => { + it('persists a unanimously clean-empty round only because it was FULLY accounted', () => { const classification = classifyContextGraphCatchupReadiness({ result: publicEmptyRoundResult(), includeSharedMemory: false, @@ -227,14 +227,19 @@ describe('context graph catch-up readiness classification', () => { synced: true, sharedMemorySynced: false, }, - // Reported ready, but NOT persisted as provenance: readiness derived from - // the absence of evidence has to be re-derived every run. See the - // two-run regression below. + // Every attempted peer answered (`failedPeers: 0`), so the empty verdict + // was taken over the whole peer set and may be written down. The same + // round with a peer unaccounted for must NOT be — see the next case. readinessPatch: { - durableVerified: false, + durableVerified: true, sharedMemoryVerified: false, }, }); + + // `synced` is a second persisted readiness bit and gates write preflight, + // so it has to carry the SAME verdict as the provenance patch. + expect(classification.statePatch?.synced) + .toBe(classification.readinessPatch?.durableVerified); }); it('re-derives an empty verdict instead of carrying it into the next run', () => { @@ -258,6 +263,10 @@ describe('context graph catch-up readiness classification', () => { }); expect(first.jobStatus).toBe('done'); expect(first.readinessPatch).toMatchObject({ durableVerified: false }); + // …and the subscription must not be marked synced either, or write + // preflight (`contextGraphRowIsWritable`: `subscribed && synced`) would + // grant durable readiness the provenance store deliberately withheld. + expect(first.statePatch?.synced).toBe(false); // Second run: metadata only, nothing proven. Feed back exactly what run one // persisted. If the empty verdict had been frozen, this would still say @@ -368,8 +377,25 @@ describe('context graph catch-up readiness classification', () => { readinessBeforeCatchup, })).toMatchObject({ jobStatus: 'done', - // Same rule as the non-legacy path: settled for this run, not frozen. + // Same rule as the non-legacy path: a fully accounted empty round is + // written down, a partial one is not. + readinessPatch: { durableVerified: true }, + }); + + // The partial-round half of that rule, on the legacy branch too. + const lossy = publicEmptyRoundResult(); + delete lossy.cleanPlaneCompletions; + lossy.diagnostics!.durable.failedPeers = 1; + expect(classifyContextGraphCatchupReadiness({ + result: lossy, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + })).toMatchObject({ + jobStatus: 'done', readinessPatch: { durableVerified: false }, + statePatch: { synced: false }, }); }); From 166d35f01ac385cd93d8ff71d0d2c9d3208eb42d Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 09:43:50 +0200 Subject: [PATCH 42/44] fix(sync): require a whole, unambiguous binding before authority MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from the merge-readiness review, all independently reproduced against the code before accepting. P1 — ambiguous or partial own `_meta` still became authority. `ownMetaConfirmsCuratorBinding` source-qualified the read correctly and then accepted membership rather than a binding: `ownCurators.includes()` plus `.some()` over creators. Two stray triples in `/_meta` earned `metadata`, and where several creator rows coexisted — the shape `createContextGraph` itself calls out as "stray creator/curator triples (e.g. from a previous build that backfilled per node)" — whichever one happened to match the candidate peer decided who spoke for the graph. Authority now requires the graph to have DECLARED itself (a canonical definition, not a mention) and the binding to be unique: exactly one curator DID, and for a wallet curator exactly one non-wallet creator peer DID matching the selected peer. Requiring a complete definition is free on a receiver rather than a new `_meta`-completeness assumption: `curator-meta-refresh` already REFUSES to install a curator snapshot missing type/access-policy/creator/curator, and the subscriber bootstrap writes type and policy but never a curator. Graphs predating that invariant lose the early stop and fall back to bounded fan-out — fail-closed, and the same trade this PR already makes for ONTOLOGY-defined graphs. P1 — a mixed clean-empty + incomplete-empty round was frozen as synced. `catchupPeerPlaneEvidence` erases an incomplete peer to an all-zero record, so a peer that answered EMPTY but never finished paging is indistinguishable from one never contacted. An explicit `complete: false` is not a transport failure, so it never reaches `failedPeers` either, and the fully-accounted gate added in `cae121181` structurally could not see it: one clean-empty peer alongside it read as unanimously empty and froze `synced: true`. `incompleteResponders` records a peer that answered without resolving, and voids the whole-round empty verdict. Pure transport failures are excluded — they are already `failedPeers`, and folding them in would pin legitimately empty graphs in a retry loop on a lossy network. P2 — a late timer could start an admission past the budget. The deadline was read only when SIZING the sleep; timers are lower bounds, so the loop could wake past `retryUntil` and admit anyway. Re-checked after the sleep. This declines to START an attempt and never interrupts one in flight. The fixture in `catchup-runner-worker-impl.test.ts` set the budget to 250 ms — exactly `CATCHUP_BACKPRESSURE_BASE_DELAY_MS` — so the first backoff was clamped to the entire budget and three tests depended on an attempt being admitted ON the deadline. Production runs 180 s against the same 250 ms base, so that ratio pinned a fixture artefact; the budget is now 900 ms and each test keeps its original intent. Every guard mutation-checked. The one worth recording: dropping the curator dedupe fails SIX tests, because `applyFact` records one declared curator twice (`pushUnique(curators)` AND `curator ??=`). A naive "exactly one" over the raw array would have rejected every real graph while passing fixtures that leave `curators` empty — so the fixtures now derive their arrays the way the projection does, and that shape is no longer optional. Refs #2006 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-cg-resolve.ts | 48 ++++++-- packages/agent/src/sync/catchup-policy.ts | 10 ++ packages/agent/test/catchup-policy.test.ts | 28 +++++ .../agent/test/cg-resolve-refresh.test.ts | 8 +- packages/agent/test/sync-policy.test.ts | 105 +++++++++++++++++- packages/cli/src/catchup-runner.ts | 39 ++++++- .../test/catchup-runner-worker-impl.test.ts | 26 ++++- .../context-graph-catchup-readiness.test.ts | 28 +++++ 8 files changed, 277 insertions(+), 15 deletions(-) diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 926744fc60..23de85cf99 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -622,9 +622,29 @@ async function ownMetaConfirmsCuratorBinding( return false; } - // The graph must name this curator itself… - const ownCurators = [own.curator, ...own.curators].filter(Boolean); - if (!ownCurators.includes(curatorDid)) return false; + // The graph must DECLARE itself here, not merely be mentioned. + // + // `/_meta` accumulates rows from several writers — durable-meta sync, + // curator refresh, gossip, per-node backfill — so the presence of a curator + // triple says only that some row arrived, not that this node holds the graph's + // canonical definition. Two stray triples were enough to earn authority. + // + // A complete definition is the cheapest honest proxy, and on a receiver it + // costs nothing: `curator-meta-refresh` REJECTS any curator snapshot missing + // type/access-policy/creator/curator before it will replace `_meta`, and the + // subscriber bootstrap writes type and policy but never a curator. So a + // curator row essentially cannot be present without them. Graphs predating + // that invariant lose the early stop and fall back to bounded fan-out. + if (!own.declared || own.accessPolicy === undefined) return false; + + // The graph must name this curator, unambiguously. + // + // Deduplicate first: `applyFact` records the SAME triple twice — it pushes to + // `curators` and also sets the `curator` scalar — so a single declared curator + // arrives here as two entries. Counting the raw array would reject every real + // graph while passing any fixture that leaves `curators` empty. + const ownCurators = new Set([own.curator, ...own.curators].filter(Boolean)); + if (ownCurators.size !== 1 || !ownCurators.has(curatorDid)) return false; const didPrefix = 'did:dkg:agent:'; const curatorIdentifier = curatorDid.slice(didPrefix.length); @@ -649,10 +669,24 @@ async function ownMetaConfirmsCuratorBinding( // ORDERS the walk, but ordering is all it may do: ending the walk means // accepting one peer's snapshot as the whole graph, and an untrusted peer's // `complete` flag says only that it served its own view. - return [own.creator, ...own.creators] - .filter((value): value is string => Boolean(value)) - .some((creatorDid) => creatorDid.startsWith(didPrefix) - && creatorDid.slice(didPrefix.length) === curatorPeerId); + // + // The binding must also be UNIQUE. Accepting "some declared creator matches + // the candidate" lets a stale or duplicated creator row — the shape + // `createContextGraph` calls out as "stray creator/curator triples (e.g. from + // a previous build that backfilled per node)" — decide which peer speaks for + // the graph, and the merged projection picks among creators in arbitrary + // order. Two candidate bindings mean no binding: rank, never settle. + const declaredPeerBindings = new Set( + [own.creator, ...own.creators] + .filter((value): value is string => Boolean(value)) + .filter((creatorDid) => creatorDid.startsWith(didPrefix)) + .map((creatorDid) => creatorDid.slice(didPrefix.length)) + // A wallet-form creator is not a peer binding; it cannot name a peer and + // must not count toward the ambiguity it would otherwise manufacture. + .filter((creatorId) => !creatorId.startsWith('0x')), + ); + if (declaredPeerBindings.size !== 1) return false; + return declaredPeerBindings.has(curatorPeerId); } export async function resolveCuratorSyncPeer( diff --git a/packages/agent/src/sync/catchup-policy.ts b/packages/agent/src/sync/catchup-policy.ts index e63448cada..d3cd44a159 100644 --- a/packages/agent/src/sync/catchup-policy.ts +++ b/packages/agent/src/sync/catchup-policy.ts @@ -283,6 +283,16 @@ export async function runCatchupPlaneWithPolicy( }); if (delayMs === undefined) return result; await wait(delayMs); + // Re-check the deadline AFTER sleeping. `nextCatchupBackpressureDelayMs` + // sized the delay against the budget that remained before the sleep, but a + // timer is a lower bound: under event-loop pressure the wake-up can land + // well past `retryUntil`, and starting a fresh admission there would spend + // scheduler capacity outside the budget this plane advertised. + // + // This declines to START an attempt; an attempt already in flight is never + // interrupted. The deliberate collateral is that capacity clearing exactly + // at or just after the deadline no longer gets one extra try. + if (now() >= retryUntil) return result; result = await run(context); } } diff --git a/packages/agent/test/catchup-policy.test.ts b/packages/agent/test/catchup-policy.test.ts index 5f6e998678..69ba515e31 100644 --- a/packages/agent/test/catchup-policy.test.ts +++ b/packages/agent/test/catchup-policy.test.ts @@ -160,6 +160,34 @@ describe('runCatchupPlanesWithPolicy', () => { expect(syncDurable).toHaveBeenCalledTimes(2); }); + it('does not start a fresh attempt when the timer wakes past the deadline', async () => { + // A timer is a LOWER bound. `nextCatchupBackpressureDelayMs` sizes the sleep + // against the budget remaining BEFORE it, so under event-loop pressure the + // wake-up can land past `retryUntil` — and starting another admission there + // spends sync-global capacity outside the budget this plane advertised. + // + // The clock below models exactly that: every sleep overruns its request. + const overshootMs = 5_000; + let nowMs = 1_000; + const clock = { + now: () => nowMs, + wait: async (delayMs: number) => { nowMs += delayMs + overshootMs; }, + }; + const syncDurable = vi.fn(async () => ({ deferredBackpressure: 1 })); + + await runCatchupPlaneWithPolicy('foreground', syncDurable, { + retry: { maxWaitMs: 1_000 }, + now: clock.now, + wait: clock.wait, + random: () => 0, + }); + + // The opening attempt, and nothing after the overrun. Without the post-sleep + // deadline check this is 2 — the loop sleeps once, wakes 5 s past a 1 s + // budget, and admits anyway. + expect(syncDurable).toHaveBeenCalledTimes(1); + }); + it('never sleeps past the retry deadline', async () => { const clock = virtualClock(); const deadlineMs = 1_000; diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index ba130c991f..0d5860a5f8 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1232,10 +1232,14 @@ describe('refreshMetaFromCurator', () => { const authoritativePeer = 'peer-from-authoritative-meta'; const preferredSyncPeers = new Map([[contextGraphId, bootstrapPeer]]); const declaredFacts = { + // A complete canonical definition, in the shape the projection produces: + // each declared fact appears as the scalar AND in its array. + declared: true, + accessPolicy: 'private', curator: 'did:dkg:agent:0x0000000000000000000000000000000000000abc', - curators: [], + curators: ['did:dkg:agent:0x0000000000000000000000000000000000000abc'], creator: `did:dkg:agent:${authoritativePeer}`, - creators: [], + creators: [`did:dkg:agent:${authoritativePeer}`], }; const agent = { preferredSyncPeers, diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 2454e52ace..5f013b6cf3 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -108,13 +108,39 @@ describe('curator sync-peer provenance', () => { curators?: string[]; creator?: string; creators?: string[]; + declared?: boolean; + accessPolicy?: string; }; + /** + * Shape a fixture the way the projection actually shapes a record. + * + * `applyFact` records ONE declared curator twice — `pushUnique(record.curators, + * o)` AND `record.curator ??= o` — so a real single-curator graph arrives with + * `curators.length === 1` and a matching scalar, i.e. two entries once the + * scalar is prepended. Fixtures that set only the scalar and leave the array + * empty do NOT look like production, and a cardinality bug that rejects every + * real graph would pass against them. Deriving the arrays here keeps every + * fixture in the production shape by construction. + */ + function projected(facts: MetaFacts) { + return { + ...facts, + curators: facts.curators ?? (facts.curator ? [facts.curator] : []), + creators: facts.creators ?? (facts.creator ? [facts.creator] : []), + }; + } + /** * `getCgMeta` is the MERGED projection (`_meta` + AGENTS + `_catalog` + * ONTOLOGY); `getOwnCgMetaFacts` is what the Context Graph declared about * itself. `ownMeta` defaults to `meta` — the ordinary case where they agree — * so any test exercising the difference has to say so out loud. + * + * `ownMeta` also carries a COMPLETE canonical definition by default + * (`declared` + an access policy), because that is what a receiver's `_meta` + * always holds: `curator-meta-refresh` refuses to install a snapshot without + * it. A test that wants the partial shape states it explicitly. */ function agentWithMeta( meta: MetaFacts, @@ -122,8 +148,12 @@ describe('curator sync-peer provenance', () => { ownMeta: MetaFacts = meta, ) { return { - getCgMeta: async () => ({ curators: [], creators: [], ...meta }), - getOwnCgMetaFacts: async () => ({ curators: [], creators: [], ...ownMeta }), + getCgMeta: async () => projected(meta), + getOwnCgMetaFacts: async () => projected({ + declared: true, + accessPolicy: 'private', + ...ownMeta, + }), discovery: { findAgents }, }; } @@ -253,6 +283,67 @@ describe('curator sync-peer provenance', () => { expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); + it('refuses authority to a graph that never declared itself', async () => { + // The reviewer's "identity-only partial own _meta": a couple of rows landed + // in `/_meta` without the canonical definition. `_meta` accumulates from + // several writers, so a curator row alone does not mean this node holds the + // graph's definition — and two stray triples must not be able to end a walk. + const declared = { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }; + const agent = agentWithMeta(declared, async () => [], { + ...declared, + declared: false, + accessPolicy: undefined, + }); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('refuses authority when the graph declares two candidate creator peers', async () => { + // `createContextGraph` calls this shape out directly — "stray creator/curator + // triples (e.g. from a previous build that backfilled per node)" — and the + // merged projection picks among creators in arbitrary order. Accepting + // whichever one happens to match the candidate peer lets a stale member + // speak for the graph. + const agent = agentWithMeta( + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creator: `did:dkg:agent:${CURATOR}`, + }, + async () => { + throw new Error('the registry fallback must not be reached here'); + }, + { + curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', + creators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWStaleMember'], + }, + ); + + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).toBe('projection'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + }); + + it('refuses authority when the graph declares two different curators', async () => { + const agent = agentWithMeta( + { curator: `did:dkg:agent:${CURATOR}` }, + async () => [], + { + curator: `did:dkg:agent:${CURATOR}`, + curators: [`did:dkg:agent:${CURATOR}`, 'did:dkg:agent:12D3KooWOtherCurator'], + }, + ); + + expect(authoritativeSyncPeerId( + await resolveCuratorSyncPeer(agent as never, new Map(), CG), + )).toBeUndefined(); + }); + it('demotes when the graph does not name that curator at all', async () => { // A curator the merged projection asserts but the graph never claimed. const agent = agentWithMeta( @@ -341,7 +432,15 @@ describe('curator sync-peer provenance', () => { // metadata confirms a curator, so the second call runs against a different // map than the first. let metaReads = 0; - const declared = { curator: `did:dkg:agent:${CURATOR}`, curators: [], creators: [] }; + // Production shape: the projection records the declared curator both as the + // scalar and in the array, alongside the canonical definition facts. + const declared = { + declared: true, + accessPolicy: 'private', + curator: `did:dkg:agent:${CURATOR}`, + curators: [`did:dkg:agent:${CURATOR}`], + creators: [], + }; const agent = { preferredSyncPeers: new Map([[CG, CURATOR]]), getCgMeta: async () => { diff --git a/packages/cli/src/catchup-runner.ts b/packages/cli/src/catchup-runner.ts index e2eec4019d..8c06daf29c 100644 --- a/packages/cli/src/catchup-runner.ts +++ b/packages/cli/src/catchup-runner.ts @@ -525,6 +525,25 @@ export interface CatchupPlaneCompletionEvidence { * {@link catchupPlaneProvenByUnanimousEmpty}. */ authorityEmptyPeers?: number; + /** + * Peers that ANSWERED this plane but whose round did not complete cleanly. + * + * Every other field here records what a peer proved. This one records what a + * peer left unresolved, and it exists because the absence of a peer from the + * positive counters is ambiguous: `catchupPeerPlaneEvidence` returns an + * all-zero record for an incomplete round, so a peer that answered EMPTY but + * did not finish paging is indistinguishable from a peer that was never + * contacted. That ambiguity is invisible to the round diagnostics too — an + * explicit `complete: false` is not a transport failure, so it never reaches + * `failedPeers`. + * + * Without it, a round of one clean-empty peer plus one incomplete-empty peer + * reads as unanimously empty. Pure transport failures are deliberately NOT + * counted here: an unreachable stranger is already `failedPeers`, and folding + * it in would pin legitimately empty graphs in a retry loop on a lossy + * network. + */ + incompleteResponders?: number; } /** The aggregate per-plane counters a whole-round verdict is allowed to consult. */ @@ -587,7 +606,15 @@ export function catchupPeerPlaneEvidence( emptyPeers: 0, authorityEmptyPeers: 0, }; - if (!plane || !catchupPlaneCompletedWithoutFailure(plane, options.complete)) return none; + if (!plane) return none; + if (!catchupPlaneCompletedWithoutFailure(plane, options.complete)) { + // The peer answered and its round was not clean. A pure transport failure is + // NOT that: we never heard from it, it is already counted in `failedPeers`, + // and treating unreachable strangers as unresolved evidence would stop a + // genuinely empty graph from ever settling on a lossy network. + const answeredButUnresolved = (plane.failedPeers ?? 0) === 0; + return answeredButUnresolved ? { ...none, incompleteResponders: 1 } : none; + } // "The host says there is nothing here." Only the curator can say it: a // response is content-free either by being wire-empty or by carrying nothing // but `_meta`, and only the metadata-resolved curator's silence about data @@ -642,6 +669,9 @@ export function addCatchupPlaneEvidence( if (peer.authorityEmptyPeers) { total.authorityEmptyPeers = (total.authorityEmptyPeers ?? 0) + peer.authorityEmptyPeers; } + if (peer.incompleteResponders) { + total.incompleteResponders = (total.incompleteResponders ?? 0) + peer.incompleteResponders; + } } /** @@ -781,6 +811,13 @@ export function catchupPlaneProvenByUnanimousEmpty( // A non-curator that has `_meta` and no data cannot tell "the graph is empty" // from "I have not synced it yet". See the note above. if ((diagnostics?.metaOnlyResponses ?? 0) > 0) return false; + // A peer that answered but did not finish paging leaves the round unresolved: + // "nobody had anything" cannot be concluded while somebody's answer is still + // half-delivered. This is not covered by the phase counters below — an + // explicit `complete: false` is neither a failure nor a timeout — and it is + // the one shape that survives `catchupPeerPlaneEvidence` erasing the peer to + // an all-zero record. + if ((completion?.incompleteResponders ?? 0) > 0) return false; // Completion evidence is PER-PEER and says explicitly whether that peer's // round was clean; `diagnostics.emptyResponses` is a raw aggregate that counts // an empty payload even when the peer's round was NOT complete. Where the diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 78a3c88e6b..361a21195e 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -29,9 +29,17 @@ import type { CatchupJobResult, CatchupRunRequest } from '../src/catchup-runner. // up — but it mutates the REAL process env, and vitest can reuse a worker // process across files in a shard. Anything loaded afterwards, including a // daemon spawned by a sibling suite, would otherwise inherit the shortened backpressure budget. +// +// The budget must stay comfortably ABOVE `CATCHUP_BACKPRESSURE_BASE_DELAY_MS` +// (250 ms). At exactly 250 ms the first backoff is clamped to the whole +// remaining budget, so the sleep ends ON the deadline and every retry in this +// file depended on the loop admitting an attempt there — which the post-sleep +// deadline check now declines. That ratio does not occur in production (the +// default budget is 180 s against the same 250 ms base), so pinning it would +// have pinned an artefact of the fixture rather than a behaviour. const previousCATCHUPBACKPRESSUREMAXWAITMS = vi.hoisted(() => { const before = process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS; - process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = '250'; + process.env.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS = '900'; return before; }); @@ -1422,8 +1430,11 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }, - sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0 }, + sharedMemory: { verifiedDataPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, incompleteResponders: 1 }, }); expect(result.diagnostics?.durable.verifiedPrivateOnlyResponses).toBe(0); }); @@ -1473,6 +1484,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); @@ -1512,6 +1526,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); @@ -1558,6 +1575,8 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedPrivateOnlyPeers: 1, emptyPeers: 0, authorityEmptyPeers: 0, + // No `incompleteResponders`: this peer COMPLETED cleanly. The counter + // must not appear merely because a plane carried no public data. }); }); @@ -1599,6 +1618,9 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) verifiedPrivateOnlyPeers: 0, emptyPeers: 0, authorityEmptyPeers: 0, + // The peer answered and did not complete cleanly: recorded so a whole-round + // empty verdict cannot be drawn over a half-delivered answer. + incompleteResponders: 1, }); }); diff --git a/packages/cli/test/context-graph-catchup-readiness.test.ts b/packages/cli/test/context-graph-catchup-readiness.test.ts index 0c8df00287..8071901332 100644 --- a/packages/cli/test/context-graph-catchup-readiness.test.ts +++ b/packages/cli/test/context-graph-catchup-readiness.test.ts @@ -242,6 +242,34 @@ describe('context graph catch-up readiness classification', () => { .toBe(classification.readinessPatch?.durableVerified); }); + it('does not settle a round where one peer answered empty but never completed', () => { + // The shape the fully-accounted check alone cannot see. Peer A completes + // empty; peer B returns an empty payload but `complete: false`. + // `catchupPeerPlaneEvidence` erases B to an all-zero record, so `emptyPeers` + // stays 1 and — because an incomplete round is NOT a transport failure — + // `failedPeers` stays 0. Both the unanimous-empty proof and the + // fully-accounted gate would therefore pass, and the graph would be frozen + // as synced on half an answer. + const mixed = publicEmptyRoundResult(); + mixed.cleanPlaneCompletions!.durable.emptyPeers = 1; + mixed.cleanPlaneCompletions!.durable.incompleteResponders = 1; + mixed.diagnostics!.durable.emptyResponses = 2; + mixed.diagnostics!.durable.failedPeers = 0; + + const classification = classifyContextGraphCatchupReadiness({ + result: mixed, + includeSharedMemory: false, + hasConfirmedMeta: true, + isPrivate: false, + readinessBeforeCatchup, + }); + + expect(classification.jobStatus).not.toBe('done'); + expect(classification.readinessPatch).toMatchObject({ durableVerified: false }); + // …and nothing opens the writeability gate either. + expect(classification.statePatch?.synced).toBe(false); + }); + it('re-derives an empty verdict instead of carrying it into the next run', () => { // The false-`done` residual: with no authoritative curator, one unrelated // empty response alongside transport-level peer failures satisfies the From e7f46dca2bccde62e05146763bd63035ab2f5633 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 10:47:46 +0200 Subject: [PATCH 43/44] fix(sync): withhold catch-up authority until a binding has a trusted writer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No resolution earns `metadata` provenance any more, so nothing this resolver returns can end a catch-up walk. Source-qualifying by GRAPH was the mistake. Reading `/_meta` alone instead of the merged projection establishes which graph HOLDS the rows, never which writer SUPPLIED them — and ordinary durable-meta catch-up admits IRI-subject descriptive metadata for the Context Graph's entity subject (`selectAdmittedMetadataIndexes` falls through for any predicate outside the control set) and inserts it verbatim into that exact graph. A contacted peer can therefore supply the rows the check reads. Three suffice: `rdf:type`, `accessPolicy`, and a `curator` DID naming itself — for a bare peer-id curator the DID IS the binding, so no creator row is even needed. Tightening the record's SHAPE could never have fixed this; completeness and uniqueness only raise the number of rows the peer must send. My own comment listed "durable-meta sync, curator refresh, gossip, per-node backfill" as `_meta` writers and then justified the proxy citing only the two validated ones — it enumerated the hole and argued past it. Authority needs a binding from a source a peer cannot write. None exists today: join approvals carry no curator signature (only the JOINER's delegation is signed), no chain record maps an agent wallet to a libp2p peer id, and both metadata "proofs" are structural checks with no signature at all. Filed as a follow-up. Consequence, stated plainly: the fan-out reduction is inert. Without an authority `authorityProven` never sets, so there is no early stop and no per-plane narrowing, and byte volume returns to the pre-PR level. What remains effective and independent of authority: the fail-closed readiness rules, the wall-clock backpressure budget, the bounded admission-source labels, and the worker-exit latch. Also collapses the walk back to ONE bounded pass when no authority is resolvable. Waves exist solely so an authority can cut the walk short; with nothing able to break the loop they save no fetch and only add a barrier between them, which would have left this PR SLOWER than the sliding window it replaced. Mutation-checked: re-granting authority from accumulated `_meta` fails 13 tests; keeping waves without an authority fails the new single-pass test. That test pins the sliding window rather than peak concurrency, because both shapes peak at the cap and a barrier is invisible to that assertion. Refs #2006 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- packages/agent/src/dkg-agent-cg-resolve.ts | 129 ++++-------------- .../agent/test/cg-resolve-refresh.test.ts | 6 +- packages/agent/test/sync-policy.test.ts | 86 +++++++----- .../cli/src/catchup-runner-worker-impl.ts | 8 +- .../test/catchup-runner-worker-impl.test.ts | 54 ++++++++ 5 files changed, 138 insertions(+), 145 deletions(-) diff --git a/packages/agent/src/dkg-agent-cg-resolve.ts b/packages/agent/src/dkg-agent-cg-resolve.ts index 23de85cf99..6c514103b7 100644 --- a/packages/agent/src/dkg-agent-cg-resolve.ts +++ b/packages/agent/src/dkg-agent-cg-resolve.ts @@ -592,103 +592,6 @@ function curatorDidNeedsRegistryResolution(curatorIdentifier: string): boolean { * was echoed back". Only the resolver knows which branch it took. */ -/** - * Does the Context Graph's OWN `/_meta` graph declare this exact - * curator→peer binding? - * - * The merged projection cannot answer this: it unions `_meta` with AGENTS, - * `_catalog` and ONTOLOGY and drops the source of each fact, so a stale or - * third-party creator declaration is indistinguishable from the graph's own. - * Catch-up authority lets ONE peer stand for a whole graph, so it needs the - * stronger statement. - * - * Returns false on any read failure — fail closed to ranking, never to authority. - */ -async function ownMetaConfirmsCuratorBinding( - agent: DKGAgent, - contextGraphId: string, - curatorDid: string, - curatorPeerId: string, - options: { signal?: AbortSignal }, -): Promise { - // Fail closed if the receiver cannot answer: an agent (or a hand-built test - // receiver) without the source-qualified reader ranks, never authorises. - if (typeof agent.getOwnCgMetaFacts !== 'function') return false; - let own; - try { - own = await agent.getOwnCgMetaFacts(contextGraphId, { signal: options.signal }); - } catch { - throwIfSyncAuthAborted(options.signal); - return false; - } - - // The graph must DECLARE itself here, not merely be mentioned. - // - // `/_meta` accumulates rows from several writers — durable-meta sync, - // curator refresh, gossip, per-node backfill — so the presence of a curator - // triple says only that some row arrived, not that this node holds the graph's - // canonical definition. Two stray triples were enough to earn authority. - // - // A complete definition is the cheapest honest proxy, and on a receiver it - // costs nothing: `curator-meta-refresh` REJECTS any curator snapshot missing - // type/access-policy/creator/curator before it will replace `_meta`, and the - // subscriber bootstrap writes type and policy but never a curator. So a - // curator row essentially cannot be present without them. Graphs predating - // that invariant lose the early stop and fall back to bounded fan-out. - if (!own.declared || own.accessPolicy === undefined) return false; - - // The graph must name this curator, unambiguously. - // - // Deduplicate first: `applyFact` records the SAME triple twice — it pushes to - // `curators` and also sets the `curator` scalar — so a single declared curator - // arrives here as two entries. Counting the raw array would reject every real - // graph while passing any fixture that leaves `curators` empty. - const ownCurators = new Set([own.curator, ...own.curators].filter(Boolean)); - if (ownCurators.size !== 1 || !ownCurators.has(curatorDid)) return false; - - const didPrefix = 'did:dkg:agent:'; - const curatorIdentifier = curatorDid.slice(didPrefix.length); - // …and for a bare peer-id DID that IS the binding, with nothing to reconcile. - if (!curatorDidNeedsRegistryResolution(curatorIdentifier)) { - return curatorPeerId === curatorIdentifier; - } - - // For a wallet-address curator, the peer must come from a creator the graph - // declared in its OWN `_meta` — not one contributed by AGENTS, `_catalog`, or - // ONTOLOGY. - // - // ONTOLOGY is excluded even though a PUBLIC graph's definition lives there, - // and local uniqueness does not rescue it: ONTOLOGY is network-replicated, so - // a node can hold an injected `DKG_CREATOR` for a subject WITHOUT holding the - // real one, and "the only creator I can currently see" would then authorise - // the attacker. That is the same local-cardinality fallacy that already makes - // the Agent Registry route non-authoritative, one graph over. - // - // So a public graph with a wallet curator earns no authority here, and its - // catch-up degrades to the previous bounded fan-out. The resolved peer still - // ORDERS the walk, but ordering is all it may do: ending the walk means - // accepting one peer's snapshot as the whole graph, and an untrusted peer's - // `complete` flag says only that it served its own view. - // - // The binding must also be UNIQUE. Accepting "some declared creator matches - // the candidate" lets a stale or duplicated creator row — the shape - // `createContextGraph` calls out as "stray creator/curator triples (e.g. from - // a previous build that backfilled per node)" — decide which peer speaks for - // the graph, and the merged projection picks among creators in arbitrary - // order. Two candidate bindings mean no binding: rank, never settle. - const declaredPeerBindings = new Set( - [own.creator, ...own.creators] - .filter((value): value is string => Boolean(value)) - .filter((creatorDid) => creatorDid.startsWith(didPrefix)) - .map((creatorDid) => creatorDid.slice(didPrefix.length)) - // A wallet-form creator is not a peer binding; it cannot name a peer and - // must not count toward the ambiguity it would otherwise manufacture. - .filter((creatorId) => !creatorId.startsWith('0x')), - ); - if (declaredPeerBindings.size !== 1) return false; - return declaredPeerBindings.has(curatorPeerId); -} - export async function resolveCuratorSyncPeer( agent: DKGAgent, /** @@ -774,14 +677,30 @@ export async function resolveCuratorSyncPeer( if (!resolved) return fromHint(); } - // Earn `'metadata'`: re-derive the binding from the Context Graph's OWN `_meta` - // graph and require it to agree. This is what makes the label mean "the graph - // said so", rather than "something in the merged projection said so". - if (provenance === 'projection') { - provenance = await ownMetaConfirmsCuratorBinding( - agent, contextGraphId, curatorDid, curatorPeerId, options, - ) ? 'metadata' : 'projection'; - } + // No route here earns `'metadata'`, so nothing this resolver returns may end a + // catch-up walk. That is deliberate, and it is a scope decision rather than an + // oversight — see #2006 and the follow-up issue. + // + // Earlier revisions re-derived the binding from the Context Graph's OWN + // `/_meta` graph, on the theory that reading one graph instead of the + // merged projection made the fact attributable to the graph itself. It does + // not. Source-qualifying by GRAPH establishes which graph holds the rows, not + // which WRITER supplied them: ordinary durable-meta catch-up admits + // IRI-subject descriptive metadata for the Context Graph's entity subject + // (`selectAdmittedMetadataIndexes` falls through for any predicate outside the + // control set) and inserts it verbatim into that exact `_meta` graph. A + // contacted peer can therefore supply the very rows the check reads — + // `rdf:type`, `accessPolicy`, `curator` — and manufacture its own authority. + // Tightening the SHAPE of the record (completeness, uniqueness) does not help: + // it only raises the number of rows the peer must send. + // + // Authority needs a binding from a source a peer cannot write: a + // curator-signed snapshot, an on-chain curator→peer edge, or the locally + // persisted join-approval record. None is available today — join approvals + // carry no curator signature, no chain record maps a wallet to a libp2p peer, + // and both metadata "proofs" are structural checks with no signature. Until + // one exists, every resolution ranks the walk and none ends it. + void curatorDid; bootstrapHints.delete(contextGraphId); return { peerId: curatorPeerId, provenance }; diff --git a/packages/agent/test/cg-resolve-refresh.test.ts b/packages/agent/test/cg-resolve-refresh.test.ts index 0d5860a5f8..ef11b26c26 100644 --- a/packages/agent/test/cg-resolve-refresh.test.ts +++ b/packages/agent/test/cg-resolve-refresh.test.ts @@ -1263,7 +1263,9 @@ describe('refreshMetaFromCurator', () => { // The same resolution through the lifecycle entry points, against the real // metadata rather than a stubbed curator: the join-approved peer ranks only - // until `_meta` names someone, and the metadata answer is authoritative. + // until `_meta` names someone. The declared answer then wins the RANKING — + // but it confers no authority, because `_meta` identifies the graph that + // holds the rows, not the writer that supplied them. const lifecycleAgent = { ...agent, preferredSyncPeers: new Map([[contextGraphId, bootstrapPeer]]), @@ -1271,6 +1273,6 @@ describe('refreshMetaFromCurator', () => { expect(await LifecycleSyncMethods.prototype.resolvePreferredSyncPeerId .call(lifecycleAgent as never, contextGraphId)).toBe(authoritativePeer); expect(await LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId - .call(lifecycleAgent as never, contextGraphId)).toBe(authoritativePeer); + .call(lifecycleAgent as never, contextGraphId)).toBeUndefined(); }); }); diff --git a/packages/agent/test/sync-policy.test.ts b/packages/agent/test/sync-policy.test.ts index 5f013b6cf3..190eb3878c 100644 --- a/packages/agent/test/sync-policy.test.ts +++ b/packages/agent/test/sync-policy.test.ts @@ -158,18 +158,16 @@ describe('curator sync-peer provenance', () => { }; } - it('reports a metadata curator as authoritative EVEN when it equals the bootstrap hint', async () => { + it('prefers a declared curator over the bootstrap hint, and consumes the hint', async () => { // The ordinary case on a healthy network: the join approval came from the - // curator, so both routes name the same peer. Deriving provenance by - // comparing the resolved id against the hint therefore reads the normal - // case as "unconfirmed hint" and never lets the catch-up walk stop — - // exactly where the early-stop optimisation is worth the most. + // curator, so both routes name the same peer. The declared route wins and + // the hint is spent — but the result RANKS the walk, it does not end it. const hints = new Map([[CG, CURATOR]]); const agent = agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }); - expect(await resolveCuratorSyncPeer(agent as never, hints, CG)) - .toEqual({ peerId: CURATOR, provenance: 'metadata' }); - // …and the resolver consumed the hint now that metadata has confirmed it. + const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); expect(hints.has(CG)).toBe(false); }); @@ -217,21 +215,25 @@ describe('curator sync-peer provenance', () => { expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); - it('authorises a wallet curator the graph binds to a peer in its OWN _meta', async () => { - // The route that keeps the early stop for V10 wallet-address curators: the - // Context Graph itself declares both the curator DID and the creator peer. + it('resolves a wallet curator from a declared creator WITHOUT granting authority', async () => { + // The graph's own `/_meta` declaring both the curator DID and the + // creator peer is the strongest statement available locally — and it is + // still not enough. Ordinary durable-meta catch-up admits descriptive rows + // for the Context Graph's entity subject and writes them into that very + // graph, so a contacted peer can supply these exact rows. Reading one graph + // instead of the merged projection identifies the GRAPH, never the WRITER. const hints = new Map([[CG, HINT]]); const declared = { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', creator: `did:dkg:agent:${CURATOR}`, }; const agent = agentWithMeta(declared, async () => { - throw new Error('an own-_meta binding must not need the registry'); + throw new Error('a declared creator must not need the registry'); }, declared); const resolved = await resolveCuratorSyncPeer(agent as never, hints, CG); - expect(resolved).toEqual({ peerId: CURATOR, provenance: 'metadata' }); - expect(authoritativeSyncPeerId(resolved)).toBe(CURATOR); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); it('demotes a creator the MERGED projection supplied but the graph did not', async () => { @@ -402,27 +404,37 @@ describe('curator sync-peer provenance', () => { expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); }); - it('keeps the deterministic metadata routes authoritative', async () => { - // The positive complement, so the guard cannot be widened into something - // that disables the early stop wholesale. Neither of these routes touches - // the registry: a bare peer-id DID, and the projected DKG_CREATOR triple. - const bareDid = agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }, async () => { - throw new Error('a bare peer-id DID must not need the registry'); - }); - expect(authoritativeSyncPeerId( - await resolveCuratorSyncPeer(bareDid as never, new Map(), CG), - )).toBe(CURATOR); - - const declared = { + it('grants authority to NO route, however well the graph declares itself', async () => { + // The invariant, stated once over every shape that has ever been proposed + // as sufficient. Each of these ranks the walk correctly; none may end it, + // because none is attributable to a writer a peer cannot impersonate. + // + // If a future change introduces a real trust anchor — a curator-signed + // snapshot, an on-chain curator→peer edge, or the locally persisted + // join-approval record — this test is where the new positive case belongs, + // NOT a relaxation of the shapes below. + const declaredWallet = { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab', creator: `did:dkg:agent:${CURATOR}`, }; - const viaCreator = agentWithMeta(declared, async () => { - throw new Error('the DKG_CREATOR route must not need the registry'); - }, declared); - expect(authoritativeSyncPeerId( - await resolveCuratorSyncPeer(viaCreator as never, new Map(), CG), - )).toBe(CURATOR); + const shapes = [ + // A bare peer-id DID curator: the DID IS the peer, nothing to reconcile. + agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }), + // A wallet curator with the creator peer declared alongside it. + agentWithMeta(declaredWallet, async () => [], declaredWallet), + // The same, resolved through the local Agent Registry instead. + agentWithMeta( + { curator: 'did:dkg:agent:0x00000000000000000000000000000000000000ab' }, + async () => [{ agentAddress: '0x00000000000000000000000000000000000000AB', peerId: CURATOR }], + ), + ]; + + for (const agent of shapes) { + const resolved = await resolveCuratorSyncPeer(agent as never, new Map(), CG); + expect(resolved.peerId).toBe(CURATOR); + expect(resolved.provenance).not.toBe('metadata'); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); + } }); it('answers ranking and authority from ONE resolution', async () => { @@ -455,19 +467,19 @@ describe('curator sync-peer provenance', () => { const resolved = await LifecycleSyncMethods.prototype.resolveSyncPeerWithProvenance .call(agent as never, CG); - expect(resolved).toEqual({ peerId: CURATOR, provenance: 'metadata' }); + expect(resolved).toEqual({ peerId: CURATOR, provenance: 'projection' }); expect(metaReads).toBe(1); // Both narrow notions are derivable from it, matching the wrappers exactly. expect(resolved.peerId).toBe(CURATOR); - expect(authoritativeSyncPeerId(resolved)).toBe(CURATOR); + expect(authoritativeSyncPeerId(resolved)).toBeUndefined(); expect(authoritativeSyncPeerId({ peerId: CURATOR, provenance: 'bootstrap-hint' })) .toBeUndefined(); expect(authoritativeSyncPeerId({ provenance: 'none' })).toBeUndefined(); }); - it('ranks on any provenance but only lets metadata settle the walk', async () => { + it('ranks on any provenance and settles the walk on none', async () => { // The two lifecycle entry points, on their real prototypes: ranking takes - // whatever peer is available, authority takes it only from metadata. + // whatever peer is available; authority takes nothing at all. const confirmedCurator = { preferredSyncPeers: new Map([[CG, CURATOR]]), ...agentWithMeta({ curator: `did:dkg:agent:${CURATOR}` }), @@ -480,7 +492,7 @@ describe('curator sync-peer provenance', () => { const authority = LifecycleSyncMethods.prototype.resolveAuthoritativeSyncPeerId; expect(await rank.call(confirmedCurator as never, CG)).toBe(CURATOR); - expect(await authority.call(confirmedCurator as never, CG)).toBe(CURATOR); + expect(await authority.call(confirmedCurator as never, CG)).toBeUndefined(); expect(await rank.call(hintOnly as never, CG)).toBe(HINT); expect(await authority.call(hintOnly as never, CG)).toBeUndefined(); diff --git a/packages/cli/src/catchup-runner-worker-impl.ts b/packages/cli/src/catchup-runner-worker-impl.ts index 3bbe9ab32d..89fc45418d 100644 --- a/packages/cli/src/catchup-runner-worker-impl.ts +++ b/packages/cli/src/catchup-runner-worker-impl.ts @@ -458,7 +458,13 @@ async function runCatchup(request: CatchupRunRequest): Promise // peer that wave would contact. const authorityFirst = prepared.authoritativePeerId !== undefined && syncCapable[0] === prepared.authoritativePeerId; - const waveSizes = CATCHUP_STOP_ON_PROOF + // Waves exist ONLY so an authority can cut the walk short. With no authority + // resolvable nothing can ever break the loop, so splitting the peer set into + // waves cannot save a single fetch — it only adds a barrier between them, + // making the round SLOWER than the single bounded pass it replaced. Fall back + // to that pass rather than paying for a stop that cannot happen. + const canStopEarly = CATCHUP_STOP_ON_PROOF && prepared.authoritativePeerId !== undefined; + const waveSizes = canStopEarly ? catchupWaveSizes( syncCapable.length, CATCHUP_MAX_CONCURRENT_PEER_SYNCS, diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 361a21195e..19c669a631 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -1043,6 +1043,60 @@ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1) expect(startOrder).toEqual(peerIds); }); + it('walks a no-authority round as ONE pass, with no barrier between waves', async () => { + // Waves exist only so an authority can cut the walk short. With no + // authoritative curator nothing can break the loop, so splitting the peer + // set into waves saves no fetch and only adds a barrier — making the round + // slower than the single bounded pass it replaced. + // + // Barriers are invisible to a peak-concurrency assertion (both shapes peak + // at the cap), so this pins the property that actually differs: with a + // sliding window a LATER peer starts while an early slow peer is still in + // flight; behind a barrier it cannot. + const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); + let slowPeerInFlight = false; + let startedDuringSlowPeer = 0; + + await runWorkerCatchup( + { contextGraphId: 'cg-no-authority-single-pass', includeSharedMemory: false }, + async (method, args) => { + switch (method) { + case 'prepareCatchup': + return { + preferredPeerId: undefined, + authoritativePeerId: undefined, + isPrivateContextGraph: false, + peerIds, + connectedPeers: peerIds.length, + }; + case 'waitForSyncProtocol': + return true; + case 'syncDurable': { + const peerId = args[0] as string; + if (peerId === 'peer-0') { + slowPeerInFlight = true; + await delay(40); + slowPeerInFlight = false; + return { ...durableResult(), complete: false }; + } + // Anything beyond the first wave-width proves the window slid. + if (slowPeerInFlight && Number(peerId.slice('peer-'.length)) >= CATCHUP_MAX_CONCURRENT_PEER_SYNCS) { + startedDuringSlowPeer += 1; + } + await delay(1); + return { ...durableResult(), complete: false }; + } + case 'finalizeCatchup': + return null; + default: + throw new Error(`unexpected invoke: ${method}`); + } + }, + ); + + expect(startedDuringSlowPeer).toBeGreaterThan(0); + }); + it('spends the single-peer opening wave only on a sync-capable curator', async () => { // A REAL authority that is offline: metadata resolved a curator, so // `authoritativePeerId` is set, but the protocol probe filters it out. The From 73bdd65221f29129b68e33ee2666655ff58b83c8 Mon Sep 17 00:00:00 2001 From: Jurij Skornik Date: Sun, 2 Aug 2026 11:36:03 +0200 Subject: [PATCH 44/44] test(sync): stop the early-stop suites claiming a behaviour production cannot reach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every worker early-stop case hands itself an `authoritativePeerId` through the stubbed `prepareCatchup` boundary, and the bridge cases stub `resolveSyncPeerWithProvenance` to return `provenance: 'metadata'`. Since `e7f46dca2` no production route produces either, so the suites were asserting an optimisation the shipped build cannot perform — and reading green as evidence for the fan-out reduction would have been reading a fabricated premise. That gap is not hypothetical: demoting authority failed six agent tests and ZERO CLI tests, because the CLI suite supplies the very value the change removes. No logic changes. Each affected file now states what its cases do and do not establish, and points at #2018, which re-enables this machinery and owns the missing piece — a case that derives the authority through the REAL resolver/projection path rather than injecting it. The tests are annotated rather than deleted because #2018 has to satisfy exactly this contract, and deleting them would remove it. Refs #2006, #2018 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SaXdkmteqStxGBpXcUPGd1 --- .../test/catchup-runner-worker-impl.test.ts | 22 +++++++++++++++++++ .../catchup-runner-worker-killswitch.test.ts | 22 +++++++++++++++++++ .../catchup-runner-worker-lifecycle.test.ts | 7 ++++++ 3 files changed, 51 insertions(+) diff --git a/packages/cli/test/catchup-runner-worker-impl.test.ts b/packages/cli/test/catchup-runner-worker-impl.test.ts index 19c669a631..d0d401fec4 100644 --- a/packages/cli/test/catchup-runner-worker-impl.test.ts +++ b/packages/cli/test/catchup-runner-worker-impl.test.ts @@ -152,6 +152,28 @@ async function runWorkerCatchup(request: CatchupRunRequest, handler: InvokeHandl }); } +/** + * SCOPE OF THESE TESTS — read before trusting a green run. + * + * Every case below hands the worker an `authoritativePeerId` through the stubbed + * `prepareCatchup` boundary. **No production resolver route currently produces + * one.** `resolveCuratorSyncPeer` was changed in `e7f46dca2` so that nothing + * earns `metadata` provenance, because a curator-to-peer binding read out of + * accumulated `/_meta` identifies the graph that HOLDS the rows, not the + * writer that SUPPLIED them — and ordinary durable-meta catch-up lets a + * contacted peer write those very rows. + * + * So these tests verify that the worker HANDLES an authority correctly IF it is + * given one. They do NOT verify that the early stop or the per-plane narrowing + * happens in the shipped build — it cannot, and byte volume is at the pre-fix + * level until #2018 lands a trusted binding. Read as end-to-end evidence for the + * fan-out reduction they would be claiming something untrue. + * + * They are kept rather than deleted because #2018 re-enables exactly this + * machinery, and deleting them would remove the contract it has to satisfy. When + * that lands, the missing piece is a case that derives `authoritativePeerId` + * through the REAL resolver/projection path instead of injecting it here. + */ describe('catchup-runner-worker-impl bounded fan-out (sync-storm mitigation C-1)', () => { it('stops the walk at the first peer that proves every requested plane', async () => { const peerIds = Array.from({ length: 20 }, (_, i) => `peer-${i}`); diff --git a/packages/cli/test/catchup-runner-worker-killswitch.test.ts b/packages/cli/test/catchup-runner-worker-killswitch.test.ts index a25e76102f..f39454ea38 100644 --- a/packages/cli/test/catchup-runner-worker-killswitch.test.ts +++ b/packages/cli/test/catchup-runner-worker-killswitch.test.ts @@ -123,6 +123,28 @@ async function runWorkerCatchup( }); } +/** + * SCOPE OF THESE TESTS — read before trusting a green run. + * + * Every case below hands the worker an `authoritativePeerId` through the stubbed + * `prepareCatchup` boundary. **No production resolver route currently produces + * one.** `resolveCuratorSyncPeer` was changed in `e7f46dca2` so that nothing + * earns `metadata` provenance, because a curator-to-peer binding read out of + * accumulated `/_meta` identifies the graph that HOLDS the rows, not the + * writer that SUPPLIED them — and ordinary durable-meta catch-up lets a + * contacted peer write those very rows. + * + * So these tests verify that the worker HANDLES an authority correctly IF it is + * given one. They do NOT verify that the early stop or the per-plane narrowing + * happens in the shipped build — it cannot, and byte volume is at the pre-fix + * level until #2018 lands a trusted binding. Read as end-to-end evidence for the + * fan-out reduction they would be claiming something untrue. + * + * They are kept rather than deleted because #2018 re-enables exactly this + * machinery, and deleting them would remove the contract it has to satisfy. When + * that lands, the missing piece is a case that derives `authoritativePeerId` + * through the REAL resolver/projection path instead of injecting it here. + */ describe('catch-up progressive walk kill-switch', () => { it('restores the full fan-out over every peer and every requested plane', async () => { const peerIds = Array.from({ length: 12 }, (_, i) => `peer-${i}`); diff --git a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts index 4b14ac7002..ab85cc427f 100644 --- a/packages/cli/test/catchup-runner-worker-lifecycle.test.ts +++ b/packages/cli/test/catchup-runner-worker-lifecycle.test.ts @@ -160,6 +160,13 @@ describe('WorkerCatchupRunner lifecycle', () => { * and observe `source` themselves, so a regression HERE would leave them green * while production early-stopped on a stale bootstrap hint or reported * foreground catch-up as `unspecified` in scheduler diagnostics. + * + * NOTE on the authoritative cases below: they stub `resolveSyncPeerWithProvenance` + * to return `provenance: 'metadata'`, which the real resolver no longer returns + * for ANY input (`e7f46dca2`). They pin that the bridge PROPAGATES an authority + * faithfully — that `authoritativeSyncPeerId` is the single definition of who + * may end a walk, and that the bridge does not invent one from a bootstrap hint. + * They are not evidence that an authority is ever produced. See #2018. */ describe('WorkerCatchupRunner agent bridge', () => { function bridgeAgent(overrides: Record = {}) {