fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006)

CHANGELOG.md

@@ -4,6 +4,30 @@ All notable changes to the DKG V10 node are documented here. The format is based
 
 ## [Unreleased]
 
+### Fixed
+
+- **One foreground Context Graph catch-up no longer pulls the whole graph from every peer, and a stranger's silence can no longer settle it as `done`** (#2006): the peer list already arrived ranked authority-first, but the ordering never became selection — every sync-capable peer got a full durable + shared-memory pull, so a 14-peer testnet fetched the same graph 5–6 times (147,246 triples for a 24,541-triple graph, ~278 MB), saturating the node-wide `sync-global` scheduler and displacing background work. Peers are now walked in escalating waves and the walk stops as soon as the **resolved curator** has settled every requested plane; fallback peers are narrowed to the planes it has not settled — which is what removes the amplification, since the peers still contacted skip the plane already served. What the curator may settle differs by plane: it settles either plane by delivering verified DATA, but it settles a plane by being EMPTY only for durable data, which it owns. Shared memory is a per-agent-address layered union contributed by many members (`PUBLIC CGs keep the union path`), so a curator holding no shared-memory rows has said nothing about the members' layers; an empty shared-memory plane is still provable, but only as a whole-round verdict once every peer has answered. Only the curator can stop the walk, because any peer's `complete` flag proves only that it served *its own* manifest — with no resolvable curator the walk degrades to the previous full fan-out and keeps unioning every peer's data. Separately, a clean **empty** response from an unrelated peer could prove a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as `done` with 1 Knowledge Asset out of 40; emptiness is now a whole-round verdict — some peer completed cleanly empty, nobody delivered graph content, and no peer engaged and then failed. Content that arrived and failed verification (`rejectedKcs`, `dataRejectedMissingMeta`) voids the verdict outright: it proves content for the graph *exists*, which outranks any peer's silence. So does a NON-curator answering `_meta` with no data — the requester itself logs "peer may have empty or pruned data graph" for that response, and without the curator present nothing can tell an empty graph from a member that has not synced it yet. Unreachable peers are deliberately not treated as evidence either way. A registered public graph that genuinely holds nothing still settles cleanly, but on its **curator's** word rather than a stranger's: such a graph still serves its own `<cg>/_meta` definition triples, so its host answers metadata-only and could never satisfy the round rule — while accepting any peer's metadata-only round would resettle this very bug, since a member holding `_meta` but no data yet is the commonest state on the network.
+- **Foreground catch-up survives local scheduler pressure instead of giving up in under a second** (#2006): the backpressure retry budget was a fixed `[100, 250, 500]` ms ladder — 850 ms total — against admitted rounds bounded by 120 s and measured `sync-global` queue waits of 87–109 s, so a refused admission always exhausted its budget before the head of the queue could clear. It is now bounded exponential backoff with jitter against an absolute per-plane wall-clock deadline (`DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, default 180 s) taken *before* the first admission attempt, so the time an attempt itself spends queued counts against the budget rather than being added to it. The timer is unreferenced so a pending backoff cannot outlive shutdown. The budget bounds how long a plane keeps **asking**; it does not preempt a round the scheduler has already accepted, which stays bounded by `SYNC_TOTAL_TIMEOUT_MS`.
+- **A dead catch-up worker no longer pins subscribe jobs at `running` forever** (#2006): `close()` terminates the Worker, which emits `'exit'` and never `'error'`, so a pending run promise was never settled — and because the runner is constructed once per daemon, every *later* subscribe hung too, with the route's dedupe handing the stuck job back on each retry. The failure is now latched and every pending and future run fails fast with a retryable status.
+
+### Changed
+
+- **`sync-global` scheduler diagnostics attribute queue pressure to a trigger** (#2006): the `operation` dimension in `GET /api/diagnostics/backpressure` and in the `[backpressure]` log records changes from the work class alone (`durable`, which merely duplicated `lane`) to `<work class>:<source>` — for example `durable:catchup-foreground` versus `durable:on-connect` or `durable:reconcile`. Both halves are closed sets, so the label space stays bounded and free of Context Graph and peer identifiers; an unrecognised source clamps to `unspecified`. Dashboards that group on `operation` for the `sync-global` scheduler will see the new values. `GET /api/sync/catchup-status` gains `result.peersNotAttempted`, the count of sync-capable peers the walk deliberately skipped.
+
+### Removed
+
+- **`CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS` and the `retryDelaysMs` option are gone from `@origintrail-official/dkg-agent`** (#2006). Both described the fixed `[100, 250, 500]` ladder, which no longer exists: delays are now derived per attempt from an exponential curve, jitter, and the remaining wall-clock budget. A compatibility alias could only have exported a schedule the node no longer follows, so a consumer would have kept compiling while reasoning about behaviour that had changed underneath it — this is called out here rather than shipped as a silent removal. Callers that tuned the ladder should use `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS`, or the injectable `retry` / `now` / `wait` / `random` seams on `runCatchupPlanesWithPolicy` for deterministic tests. `retryDelaysMs` is retained on the options type as `never`, so a caller that still sets it fails to compile rather than having it silently ignored, and both it and the removed export are rejected at RUNTIME too — an ignored `retryDelaysMs: [10]` would otherwise turn an intended 10 ms schedule into a wait of up to the full budget, measured at 41 retry attempts and 180,000 ms of blocking against the old ladder's 3 attempts and 30 ms.
+
+  **Node operators need do nothing.** The node ships as one unit — the CLI depends on the agent as `workspace:*`, so every package moves to the same version on upgrade and no node holds a stale caller. This removal is only visible to code OUTSIDE this repository that installs `@origintrail-official/dkg-agent` from npm and calls the catch-up retry policy directly, which is an internal sync-scheduler knob rather than part of the SDK surface. Anything that does hit it gets an immediate error naming the replacement, not a silent behaviour change.
+
+### Operator knobs
+
+| Variable | Default | Effect |
+| --- | --- | --- |
+| `DKG_CATCHUP_STOP_ON_PROOF` | on | Set to `0`/`false`/`no`/`off` to restore the pre-#2006 full fan-out: every sync-capable peer, both requested planes, no early stop. The escape hatch for the deliberate tradeoff that foreground catch-up may land the curator's snapshot rather than the union of every peer's. |
+| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend waiting for local `sync-global` capacity. An explicit `0` disables retries; a blank value is treated as unset. |
+| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Unchanged. Caps in-flight per-peer sync rounds and now also caps the widest escalation wave. |
+
 ## [10.0.11] - 2026-07-30
 
 A focused stability release on two paths a busy node exercises constantly. One external `/api/query` read could amplify into a planner-stalling query that starved every other subsystem: a caller that had already constrained `GRAPH ?g` to a handful of verified partitions was rewritten with a second `VALUES ?g` carrying the entire allow-list, expanding a 3 KB query to roughly 24 KB and occupying the store for minutes, cascading into queue-wait timeouts across promotion, gossip validation, SWM catch-up, and durable sync. External reads now run on the store scheduler's background lane, a disconnected caller's store work is cancelled instead of orphaned, and the redundant graph rewrite is elided. Separately, the `dkg integration` CLI is brought back into line with the registry's published JSON Schema, which its parser had drifted *stricter* than — so no `manual` entry was readable at all, in either the CLI or the node dashboard's integrations sidebar. The dashboard database stays at 31 — no migration. **No smart-contract changes — no deployment required** (no Solidity source, ABI, or deployment-registry changes since 10.0.10).

docs/use-dkg/backpressure-observability.md

@@ -69,6 +69,43 @@ The first registered sources are:
   `normal`, and `background` lanes;
 - `sync-global`: the process-wide sync admission queue and its sync lanes.
 
+### Attributing `sync-global` pressure to a trigger
+
+The `lane` of a `sync-global` entry says *what kind of work* is queued
+(`durable`, `changelog`, `shared_memory`, `swm_recovery`), but every trigger
+funnels into the same few lanes. Its `operation` label therefore pairs the
+collapsed work class with the **admission source** — the trigger that enqueued
+it — as `<work class>:<source>`:
+
+| Source | Trigger |
+| --- | --- |
+| `catchup-foreground` | explicit Context Graph catch-up (`POST /api/context-graph/subscribe`) |
+| `catchup-background` | automatic post-approval / reconcile catch-up |
+| `on-connect` | sync-on-connect after a peer dial |
+| `reconcile` | the periodic sync reconciler |
+| `vm-recovery` | foreground repair of specific missing Knowledge Assets |
+| `swm-recovery` | curator-targeted shared-memory recovery |
+| `unspecified` | a caller that did not declare an origin |
+
+### Tuning foreground catch-up
+
+Two knobs govern the foreground Context Graph catch-up that most often shows up
+as `catchup-foreground` pressure. Both are read once at daemon start.
+
+| Variable | Default | Effect |
+| --- | --- | --- |
+| `DKG_CATCHUP_STOP_ON_PROOF` | on | The catch-up walks peers in escalating waves and stops once the resolved curator has settled every requested plane. Set to `0`, `false`, `no`, or `off` to restore the previous behaviour: every sync-capable peer, both requested planes, no early stop. Use this if a graph ever lands short — foreground catch-up optimises for one authoritative payload, while breadth remains the background reconcile lane's job. |
+| `DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS` | `180000` | Wall-clock budget one foreground plane may spend being **refused** by local `sync-global` admission before the job reports a retryable `deferred`. Measured from before the first attempt, so an attempt's own queue time counts against it. It does not cancel a round the scheduler has already accepted — that one is doing real work and is bounded by `SYNC_TOTAL_TIMEOUT_MS`. The default sits above both a full head-of-line round (120 s) and the queue waits that motivated it. An explicit `0` disables retries; a blank value is treated as unset. |
+| `DKG_CATCHUP_MAX_CONCURRENT_PEERS` | `4` | Caps in-flight per-peer sync rounds, and therefore the widest escalation wave. Raising it above the `sync-global` queue depth lets a single catch-up saturate the scheduler against itself. |
+
+So `{"operation":"durable:catchup-foreground","count":4,"oldestAgeMs":109000}`
+in a `queuedOperations` summary reads as "four explicit catch-up durable
+admissions are queued, the oldest for 109 seconds", and the matching
+`activeOperations` entry gives the same view for admitted work. Both halves are
+closed sets, so the label space stays bounded (5 × 7) and, as before, no Context
+Graph id or peer id ever reaches a metric, log line, or diagnostics response —
+an unrecognized source is clamped to `unspecified`.
+
 Other schedulers can extend `ObservableScheduler` and call its protected
 lifecycle methods at their existing admission boundaries. They keep complete
 ownership of policy.

packages/agent/src/context-graph-meta-projection.ts

@@ -128,6 +128,37 @@
   DKG_ONTOLOGY.DCT_ACCESS_RIGHTS,
 ]);
 
+/**
+ * A `ContextGraphMetaRecord` with no facts loaded yet.
+ *
+ * Shared by every reader so a field added to the record cannot be initialized
+ * in one loader and forgotten in another.
+ */
+function emptyContextGraphMetaRecord(
+  contextGraphId: string,
+  uri: string,
+): ContextGraphMetaRecord {
+  const isSystem = (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId);
+  return {
+    id: contextGraphId,
+    uri,
+    declared: isSystem,
+    isSystem,
+    creators: [],
+    curators: [],
+    allowedPeers: [],
+    allowedAgents: [],
+    participantAgents: [],
+    participantIdentityIds: [],
+    revokedAgents: [],
+    delegations: [],
+    subGraphs: [],
+    hasAgentGate: false,
+    hasPeerGate: false,
+    hasLegacyParticipantGate: false,
+  };
+}
+
 export class ContextGraphMetaProjection {
   private readonly entries = new Map<string, ProjectionEntry>();
 
@@ -312,6 +343,50 @@
     return (await this.store.listGraphs(options)).filter((graphUri) => graphUri.startsWith(prefix));
   }
 
+  /**
+   * Facts the Context Graph declared about ITSELF in its own `_meta` graph.
+   *
+   * `get()` deliberately unions `_meta`, AGENTS, `_catalog` and ONTOLOGY under
+   * first-wins precedence, which is right for privacy and listing reads — an
+   * AGENTS-only declaration can legitimately mark a graph private. It is NOT
+   * right for deciding who speaks for the graph: the merged record discards
+   * WHICH graph supplied each fact, so a creator contributed by AGENTS or
+   * `_catalog` (both of which carry THIRD-PARTY assertions — other agents'
+   * self-declarations and peer-fetchable catalog records) is indistinguishable
+   * from one the Context Graph declared about itself.
+   *
+   * Only `<cg>/_meta` is read. ONTOLOGY is deliberately NOT included even
+   * though a PUBLIC graph writes its definition there
+   * (`defGraph = isCurated ? cgMetaGraph : ontologyGraph`): ONTOLOGY is
+   * network-replicated, so any node can assert a `DKG_CREATOR` for a subject,
+   * and a row being the only one currently visible LOCALLY proves nothing
+   * about what the network holds. Requiring local uniqueness there would
+   * repeat, one graph over, the same local-cardinality fallacy that makes the
+   * Agent Registry route non-authoritative.
+   *
+   * The consequence is deliberate, and is a real cost: a public graph whose
+   * identity facts live only in replicated ONTOLOGY has NO locally trustworthy
+   * binding, so it earns no authority and its catch-up degrades to the previous
+   * bounded fan-out. That is today's behaviour rather than a regression — the
+   * fan-out reduction is earned by graphs that declare their own binding, and
+   * settling a graph on an unverifiable claim is the worse trade.
+   *
+   * Catch-up authority needs this distinction (issue #2006).
+   */
+  async getOwnMetaFacts(
+    contextGraphId: string,
+    options: QueryOptions = {},
+  ): Promise<ContextGraphMetaRecord> {
+    const uri = contextGraphDataUri(contextGraphId);
+    const metaGraph = contextGraphMetaGraphUri(contextGraphId);
+    assertSafeIri(uri);
+    assertSafeIri(metaGraph);
+
+    const record = emptyContextGraphMetaRecord(contextGraphId, uri);
+    await this.loadContextGraphFacts(metaGraph, uri, record, options);
+    return record;
+  }
+
   private async rebuild(contextGraphId: string, options: QueryOptions): Promise<ContextGraphMetaRecord> {
     const uri = contextGraphDataUri(contextGraphId);
     const ontologyGraph = contextGraphDataGraphUri(SYSTEM_CONTEXT_GRAPHS.ONTOLOGY);
@@ -330,24 +405,7 @@
     assertSafeIri(metaGraph);
     assertSafeIri(catalogGraph);
 
-    const record: ContextGraphMetaRecord = {
-      id: contextGraphId,
-      uri,
-      declared: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId),
-      isSystem: (Object.values(SYSTEM_CONTEXT_GRAPHS) as string[]).includes(contextGraphId),
-      creators: [],
-      curators: [],
-      allowedPeers: [],
-      allowedAgents: [],
-      participantAgents: [],
-      participantIdentityIds: [],
-      revokedAgents: [],
-      delegations: [],
-      subGraphs: [],
-      hasAgentGate: false,
-      hasPeerGate: false,
-      hasLegacyParticipantGate: false,
-    };
+    const record = emptyContextGraphMetaRecord(contextGraphId, uri);
 
     // Authoritative (local, fully trusted) sources first, meta-first so its
     // scalars win via first-wins (`??=`) precedence. The floor-filtered `_catalog`
@@ -375,7 +433,7 @@
       `SELECT ?delegation ?predicate ?object WHERE {
         GRAPH <${metaGraph}> {
           ?delegation <${DKG_ONTOLOGY.DKG_DELEGATION_AGENT}> ?delegatedAgent .
           ?delegation ?predicate ?object .
           VALUES ?predicate {
             <${DKG_ONTOLOGY.DKG_DELEGATION_AGENT}>
             <${DKG_ONTOLOGY.DKG_ALLOWED_DELEGATEE_PEER}>
@@ -518,7 +576,7 @@
     const result = await this.store.query(
       `SELECT ?subGraph ?name ?createdBy ?createdAt ?description WHERE {
         GRAPH <${metaGraph}> {
           ?subGraph ?typePred ?subGraphType ;
                     ?namePred ?name ;
                     ?createdByPred ?createdBy .
           VALUES ?typePred { <${DKG_ONTOLOGY.RDF_TYPE}> }
@@ -526,11 +584,11 @@
           VALUES ?namePred { <${DKG_ONTOLOGY.SCHEMA_NAME}> <${LEGACY_SCHEMA_NS}name> }
           VALUES ?createdByPred { <${DKG_NS}createdBy> <${LEGACY_DKG_NS}createdBy> }
           OPTIONAL {
             ?subGraph ?createdAtPred ?createdAt .
             VALUES ?createdAtPred { <${DKG_NS}createdAt> <${LEGACY_DKG_NS}createdAt> }
           }
           OPTIONAL {
             ?subGraph ?descriptionPred ?description .
             VALUES ?descriptionPred { <${DKG_ONTOLOGY.SCHEMA_DESCRIPTION}> <${LEGACY_SCHEMA_NS}description> }
           }
         }