Skip to content

fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006) - #2007

Merged
Jurij89 merged 44 commits into
testnet-canaryfrom
fix/2006-catchup-peer-selection
Aug 2, 2026
Merged

fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006)#2007
Jurij89 merged 44 commits into
testnet-canaryfrom
fix/2006-catchup-peer-selection

Conversation

@Jurij89

@Jurij89 Jurij89 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Important

Scope change at e7f46dca2 — the fan-out reduction is NOT in this PR.

Ending a catch-up walk early requires knowing which peer is the Context
Graph's curator, and no source available today can establish that: reading the
graph's own <cg>/_meta identifies the graph that HOLDS the rows, not the
writer that SUPPLIED them, and ordinary durable-meta catch-up lets any
contacted peer write those very rows. Rather than ship a forgeable early stop
— which would be the same false-done class this issue exists to remove — no
resolver route now grants authority, so authorityProven never sets, there is
no early stop and no per-plane narrowing, and byte volume stays at the
pre-fix level
.

This PR therefore ships the correctness and observability half of #2006:
fail-closed readiness, the wall-clock backpressure budget, queue-origin
attribution, and the worker-exit latch. The reported false-done symptom is
fixed. The amplification (D1) is refiled as #2018, whose first task is the
trusted curator-to-peer binding this needs.

The walk machinery is retained rather than deleted because #2018 re-enables
exactly it; the walk collapses to a single bounded pass while no authority can
resolve, so it costs nothing versus the previous behaviour.

Foreground Context Graph catch-up (the POST /api/context-graph/subscribe job) pulled the whole graph from every sync-capable peer. On a 14-peer testnet that is 5–13 redundant full payloads — 147,246 fetched triples for a 24,541-triple graph, ~278 MB — which saturates the node-wide sync-global scheduler (2 inflight / 4 queued) and displaces background work. Separately, a clean empty response from an unrelated peer proved a public plane ready, so a run that fetched 122,705 triples and failed five phases settled as done with 1 KA out of 40.

  • Progressive peer walk, gated on an authority (present but INERT — see the notice above; no route currently produces an authority, so the walk runs as a single bounded pass). The peer list already arrived ranked authority-first (orderCatchupPeers: preferred/curator → known cores → rest), but that ordering never became selection. Peers are now contacted in escalating waves (1 → 2 → 4, capped by the existing DKG_CATCHUP_MAX_CONCURRENT_PEERS), fallback peers are narrowed to the planes the curator has not settled, and the walk stops once every requested plane is settled. Only the metadata-resolved curator can settle a plane — any peer's complete flag proves only that it served its own manifest — so with no resolvable curator nothing is authority-proven and the walk degrades to the previous full bounded fan-out. The single-peer opening wave is taken only when a sync-capable curator is actually first in the ranked list. The stop decision is evaluated at the END of each wave against the round's accumulated diagnostics, so a contradiction raised by any peer in that wave is visible regardless of arrival order. DKG_CATCHUP_STOP_ON_PROOF=0 restores the previous full fan-out.

    Most of the saving would come from per-plane narrowing, not from the break — but neither is reachable in the shipped build, since narrowing is gated on the same authority. Once the curator settles durable, every fallback peer takes the durable: null branch — which is what removes the 147,246-triple / ~278 MB re-pull — even when the walk keeps going for shared memory.

  • Fail-closed empty proof. An empty response cannot distinguish a peer that hosts an empty graph from one that never heard of it — an unknown CG has no access policy, so the responder authorizes the request and its CG-scoped queries return zero rows. The requester emits emptyResponses only when both phase payloads are empty, so an empty answer can never carry hosting evidence. Readiness is therefore proven by one of three named modes, in order of strength — see the table below.

  • Wall-clock backpressure budget. The foreground retry ladder was a fixed [100, 250, 500] — 850 ms total — against admitted rounds bounded by SYNC_TOTAL_TIMEOUT_MS (120 s) and measured sync-global queue waits of 87–109 s, so a refused admission always exhausted its budget long before the head of the queue could clear. Replaced with bounded exponential backoff plus jitter against an absolute per-plane deadline (DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, default 180 s). Waiting costs a timer and no work; the sleep is unref()ed so a pending backoff cannot outlive agent.stop(). Cancellation needs no new plumbing — an aborted admission raises an AbortError, not a SyncBackpressureBusyError, so it never sets deferredBackpressure and the loop exits on its next check.

  • Queue-origin observability. sync-global admissions now carry a bounded SyncAdmissionSource, so the operation dimension reads durable:catchup-foreground instead of duplicating lane. The existing /api/diagnostics/backpressure snapshot and [backpressure] log records already expose per-operation counts plus oldestQueuedAgeMs / oldestActiveAgeMs, so pressure can now be attributed to a trigger — explicit catch-up vs sync-on-connect vs reconcile — with no core telemetry change. Deliberately not added to the shared SchedulerPressureTracker histograms: that tracker also serves the store scheduler, which passes ~192 raw source literals, so an attribute there would explode cardinality. Unknown origins clamp to unspecified at runtime, so a value crossing the worker RPC boundary cannot widen the label space or leak a Context Graph / peer identifier.

  • Worker exit safety (pre-existing, amplified by longer walks). close() terminates the Worker, which emits 'exit' and never 'error', so a pending run() promise was never settled and the fire-and-forget subscribe job stayed running with no finishedAt. Because the runner is constructed once per daemon and postMessage to a dead worker neither throws nor delivers, every later subscribe hung too — and the route's dedupe handed that stuck job back on each retry. The failure is now latched, so pending and future runs both fail fast with a retryable status.

Readiness proof model, as it now stands

Twenty-five review rounds moved this materially. This table is the current
behaviour; the chronological round log further down is history, not the model.

Mode Proves a plane when Plane scope
catchupPlaneProvenByData some peer cleanly completed carrying verified content (including a verified private-only V2 response) both
catchupPlaneProvenByAuthorityHostedEmpty the metadata-resolved curator hosted the graph and carried no data durable only
catchupPlaneProvenByUnanimousEmpty a whole round in which somebody answered cleanly empty and nobody had anything both

Four asymmetries carry the correctness, and each exists because the naive
symmetric rule was shown to be wrong — in every case by a reviewer finding, or
in one case by my own sweep, rather than by design:

  1. Only the curator's emptiness counts, and only on durable. <cg>/_meta
    carries the Context Graph's own definition triples, so a curator serving them
    proves it hosts the graph. Public shared memory is a per-agent-address layered
    union (<swm>/<addr>/<n>) contributed by many members — planSharedMemorySyncContextGraphs
    states it outright: "PUBLIC CGs keep the union path" — so a curator holding
    no SWM rows has said nothing about the members' layers.

  2. Content that exists, in any form, beats silence. Verified data, data
    fetched anywhere in the round, and content that arrived and failed
    verification (rejectedKcs, dataRejectedMissingMeta) all void an empty
    verdict. A non-curator answering _meta with no data voids it too — that is
    the commonest state on the network (a member that has not synced yet) and
    accepting it would resettle Catch-up fan-out overloads sync-global queue and can report incomplete graphs as done #2006 itself.

  3. A silent CURATOR voids the verdict; a silent stranger does not.
    authorityUnanswered records that a resolvable curator was selected and never
    cleanly answered. Voiding on failedPeers instead would also kill the verdict
    when no curator is resolvable at all — the state where mode 2 structurally
    cannot fire — pinning a legitimately empty public graph at unreachable
    behind one unreachable stranger.

  4. Only a DETERMINISTIC curator resolution may end the walk. <cg>/_meta names
    the curator either as a libp2p peer id directly (legacy) or as a wallet address
    (V10). The wallet route prefers the projected DKG_CREATOR triple, and falls
    back to an agent-registry lookup that the code has always documented as
    arbitrary when several agents register the same wallet. That was harmless while
    this only ranked the walk; it is not, now that it can end one. An ambiguous
    match therefore carries 'registry' provenance — it ranks, but only
    'metadata' is authoritative. A unique registration stays 'metadata', since
    that is a deterministic binding.

    Provenance Source May end the walk
    metadata <cg>/_meta directly, DKG_CREATOR, or a UNIQUE registry match yes
    registry wallet matched >1 agent registration — arbitrary pick no
    bootstrap-hint authenticated join-approval hint; can be stale no
    none nothing resolved n/a

Both empty modes are additionally gated, and these are the clauses that close the
reported #2006 round rather than the asymmetries above:

  • Private planes are never proven by emptiness at all — both modes return
    false outright when isPrivate, since an authorized-but-filtered response is
    indistinguishable from an empty one on this side of the wire.
  • A whole-round verdict additionally requires zero failedPhases,
    timedOutPhases, deniedPhases and deferredBackpressure. The reported run
    had five failed phases alongside 122,705 fetched triples; either clause kills
    it on its own.
  • Scope note on the two voiders. The non-curator metadata-only voider and
    authorityUnanswered apply to the whole-round mode only — neither can
    override the curator's own hosted-empty proof, which is voided solely by
    evidence that content exists. And authorityUnanswered keys on a
    'metadata'-provenance curator: a 'registry' peer is never handed to the
    worker as an authority, so its silence is not decisive either.

Changed contracts fail loudly, never silently: the removed retryDelaysMs,
the reshaped positional admission arguments, and an injected wait without a
paired now all throw rather than degrade. Each was measured degrading before
being made to fail — the last one spun 6,873,671 times in a 2-second budget.

Deliberate tradeoff

Even the curator's complete flag proves it served its own manifest, not that the manifest was network-complete (durable-sync.ts:452 compares the fetched offset against manifestRowCount, derived from the metadata that peer just sent), and SharedMemorySyncResult has no completion flag at all. Foreground catch-up is therefore optimised for one fast authoritative payload; breadth and eventual convergence remain the background reconcile lane's job, which this PR deliberately leaves fanning out. Gating on the curator bounds the exposure to "the curator's view of its own graph"; the kill-switch exists so even that can be reverted operationally without a redeploy.

Scope note: the production foreground runner is catchup-runner-worker-impl.ts and nothing else — createInlineCatchupRunner has no production caller (benchmark scripts only), and every other caller of syncContextGraphFromConnectedPeers defaults to mode: 'background'. The empty-masking defect was CLI-only; the agent-side promotion rule already required inserted data.

Related

  • Fixes Catch-up fan-out overloads sync-global queue and can report incomplete graphs as done #2006
  • Builds on feat(observability): unify scheduler backpressure diagnostics #2003 (backpressure diagnostics) — the new operation label flows straight into feat(observability): add sync pressure and source-cost flame graphs #2005's Grafana flame graph, which already groups by scheduler / lane / operation. No overlap in files.
  • Follow-up filed: Extract the catch-up readiness proof model out of catchup-runner.ts #2008 — extract the readiness proof model out of catchup-runner.ts, collapse the parallel per-plane accumulators, and type the worker RPC protocol. Raised across review rounds 9–12 and declined here on scope, not merit; this PR deliberately leaves it cheap (the model is dependency-free, and cleanPlaneCompletions now uses the shared evidence type rather than a duplicate).
  • Follow-ups worth filing (out of scope here):
    • Repeat-sync cost. A completed round deletes its checkpoint (durable-sync.ts:311) and materializeVerifiedGraphScopedAsset re-applies an equal version, so every sync re-downloads and re-writes the whole graph. After the fan-out is cut, this is the remaining bulk of the 278 MB.
    • Completeness oracle. Comparing against chain.getContextGraphKCCount(cgId) or the locally-unioned <cg>/_meta before declaring a plane proven would remove the subset tradeoff above.
    • Harness (separate repo dkg-blackbox-harness): tie per-request status deadlines to the remaining step/campaign deadline and treat transient poll failures as retryable. The issue's harness section is unchanged by this PR.

Diagrams

Foreground catch-up peer selection

Before:

sequenceDiagram
    participant Route as subscribe route
    participant Worker as catch-up worker
    participant P0 as peer-0 (curator)
    participant Pn as peer-1..13
    Route->>Worker: run(cg, includeSharedMemory)
    Worker->>P0: durable + SWM (full graph)
    Worker->>Pn: durable + SWM (full graph) x13
    P0-->>Worker: complete, verified data
    Pn-->>Worker: 5 x full payload, 5 x failed phase, 5 x clean empty
    Worker-->>Route: 147k fetched triples, emptyPeers > 0
    Route-->>Route: status = done (1 KA of 40)
Loading

After:

sequenceDiagram
    participant Route as subscribe route
    participant Worker as catch-up worker
    participant P0 as peer-0 (curator)
    participant Pn as peer-1..13
    Route->>Worker: run(cg, includeSharedMemory)
    Worker->>P0: wave 1 - durable + SWM (P0 is the resolved curator)
    P0-->>Worker: complete, verified data (both planes)
    Worker-->>Route: 24.5k fetched triples, peersNotAttempted = 13
    Note over Worker,Pn: no curator, or curator did not prove:<br/>waves 2+ walk every peer as before
    Route-->>Route: status = done (40 KAs)
Loading

The diagram shows the curator answering with verified data on both planes,
which is when the walk genuinely stops early. When shared memory is requested
and genuinely empty — the common case, since includeSharedMemory defaults to
true — the shared plane can only be settled by curator data, so the walk
usually runs every wave. The amplification fix still holds there: once the
curator settles durable, each remaining peer takes the durable: null branch,
so the expensive plane is pulled once regardless.

Readiness verdict for an empty response

Before:

sequenceDiagram
    participant Worker as catch-up worker
    participant Classifier as readiness classifier
    participant Job as subscribe job
    Worker->>Classifier: emptyPeers=5, fetchedData=122705, failedPhases=5
    Classifier-->>Classifier: !isPrivate && emptyPeers > 0
    Classifier-->>Job: durableVerified = true
    Job-->>Job: status = done
Loading

After:

sequenceDiagram
    participant Worker as catch-up worker
    participant Classifier as readiness classifier
    participant Job as subscribe job
    Worker->>Classifier: emptyPeers=5, fetchedData=122705, failedPhases=5
    Classifier-->>Classifier: verified data? no
    Classifier-->>Classifier: unanimous clean empty? no (content fetched, phases failed)
    Classifier-->>Job: durableVerified = false
    Job-->>Job: status = unreachable (retryable)
Loading

Files changed

32 files, +4,705 / −355 against origin/testnet-canary. 12 production source
files; 16 test files, 2 vitest lane configs, CHANGELOG and one doc.

File What
packages/cli/src/catchup-runner-worker-impl.ts Escalating-wave peer walk; per-plane narrowing; curator-only stop evaluated per WAVE against round diagnostics; the curator's own evidence tracked apart from the round total; authorityUnanswered; peersNotAttempted
packages/cli/src/catchup-runner.ts The proof model — catchupPeerPlaneEvidence (plane-discriminated), emptyVerdictContradicted, and the three catchupPlaneProven* predicates behind catchupPlaneReady; Worker RPC payloads typed on both ends (the method/arity protocol itself is
deferred to #2008); 'exit'/'error' failure latch; clamps the untrusted admission source at the worker edge
packages/cli/src/context-graph-readiness.ts Consumes the shared predicates; cleanCompletionHasResponse lists every evidence carrier so a new one cannot be added to the model and omitted from the pre-readiness gate
packages/agent/src/sync/catchup-policy.ts Wall-clock deadline-aware backoff with jitter replacing the fixed ladder, deadline taken BEFORE the first attempt; unref()ed sleep; rejects the removed retryDelaysMs and an unpaired wait/now seam
packages/agent/src/sync/catchup-concurrency.ts catchupWaveSizes; resolveCatchupStopOnProof + the DKG_CATCHUP_STOP_ON_PROOF kill-switch
packages/agent/src/sync/policy.ts Closed SyncAdmissionSource set + runtime normalizeSyncAdmissionSource clamp
packages/agent/src/sync/backpressure.ts operation label becomes <work class>:<source>; source threaded into the queue payload
packages/agent/src/dkg-agent-lifecycle.ts resolveSyncPeerWithProvenance (one resolution, both notions); source typed as the closed union and set at catch-up, on-connect, reconcile, changelog, VM recovery and SWM recovery; rejects the pre-#2006 positional admission shape
packages/agent/src/dkg-agent-cg-resolve.ts resolveCuratorSyncPeer returns { peerId, provenance } from the branch it took — comparing ids after resolution could not tell a confirmed curator from an echoed hint. Four-valued provenance: an AMBIGUOUS wallet-registry match ranks but cannot end the walk. authoritativeSyncPeerId is the single definition of who may
packages/agent/src/index.ts Publishes what the CLI worker genuinely needs; retry-policy test seams removed from the root
packages/cli/src/api-client.ts, cli-helpers.ts Surface peersNotAttempted in catch-up status output
CHANGELOG.md, docs/use-dkg/backpressure-observability.md Operator-facing behaviour, the <work class>:<source> label, and the removal's migration path
tests (13 files) Rewrote assertions that encoded the bug; added walk, wave-size, proof-predicate, deadline, origin-clamp, provenance, bridge-handoff and worker-lifecycle coverage, plus an enforced .typecheck.ts for the removed API

Review rounds

Chronological log of 24 review rounds. The behaviour model above is the current
state — several entries here describe rules that later rounds replaced. Kept for
auditability, including two corrections where I reported something inaccurately.

Round 1 (otReviewAgent, 3 🔴 / 5 🟡) — 4 applied in d3317b0bf + 4c02f87e5, 4 answered in thread:

Finding Outcome
🔴 Early stop treats one peer's data as proof of the whole catch-up Applied. Both the early stop and the per-plane narrowing are now gated on proof from the resolved curator; with no curator the walk degrades to the previous full fan-out. Regression test added: three peers all returning clean complete data, no preferredPeerId ⇒ all three synced, peersNotAttempted === 0.
🔴 Default backpressure budget (60 s) is below the 87–109 s waits it must survive Applied. Raised to 180 s — above both the observed backlog and the 120 s head-of-line round. Two tests: the constant is pinned > 120_000 and > 109_000, and a virtual clock keeps retrying past a 90 s capacity clear.
🔴 Removing CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS breaks consumers Declined. The constant was the policy (a fixed three-sleep ladder); nothing honours it now, so an alias would compile while describing a schedule the node no longer follows. Workspace-internal barrel; the removal is what forced both in-tree consumers to be rewritten against the new semantics. Offered a CHANGELOG entry instead.
🟡 Kill-switch path is not verified Applied. New catchup-runner-worker-killswitch.test.ts boots the worker with DKG_CATCHUP_STOP_ON_PROOF=0 and a curator that proves both planes on wave 1, and asserts the full fan-out is restored under the existing concurrency bound.
🟡 Admission metadata added as another positional optional Applied. runContextGraphSyncWithBackpressure now takes a named admission object; the SWM-recovery call site reads { source: 'swm-recovery' } instead of undefined, undefined, 'swm-recovery'.
🟡 Closed admission-source labels modeled as raw strings Answered + documented. Deliberate: source?: string sits exactly at the surfaces that can be reconstructed from a postMessage payload across the Worker RPC, and is clamped once by normalizeSyncAdmissionSource in acquire; every layer past the clamp is typed SyncAdmissionSource. 4c02f87e5 says so at each of the three option surfaces.
🟡 Catch-up internals leaking through the agent public barrel Deferred. The barrel already exports this category for this reason, with the rationale written above the block (mapWithConcurrency / CATCHUP_MAX_CONCURRENT_PEER_SYNCS, so the CLI Worker need not deep-import dist/). An internal subpath is a packaging change (exports map, build outputs, publish config) that should be reviewed on its own merits, not as a rider on a sync-path fix.
🟡 Progressive state folded into one procedural accumulator Deferred. Right end state, wrong PR: restructuring the accumulator that produces the readiness evidence, inside the change that alters what that evidence means, would put a behavioural fix and a structural rewrite behind one revert. The seam exists — one named predicate for the wave decision, one exported predicate pair shared by the walk and the classifier, pure wave arithmetic with its own tests.

Round 2 (otReviewAgent, 2 🟡) — both applied in def08c3a5:

Finding Outcome
🟡 Skipped-plane model built on any Applied. PeerRound now carries `CatchupDurableResult
🟡 Admission source stringly typed through the lifecycle Applied — a better split than the one I defended in round 1. runContextGraphSyncWithBackpressure is now the single normalization point; everything past it carries SyncAdmissionSource. The clamp in acquire stays as defence in depth and is still pinned by a deliberately bad cast.

Round 3 (independent adversarial pass over the final diff, 4 confirmed blockers) — all applied in 216bcbe5a. Three were introduced by this PR:

Finding Outcome
🔴 The empty verdict was effectively unreachable Applied. fetchedMetaTriples === 0 can essentially never hold for a hosted graph — registration writes definition triples into <cg>/_meta — so a legitimately empty public graph would have become permanently unreachable. That clause and the failedPeers clause are gone; positive tests now pin that neither metadata nor unreachable peers void the verdict.
🔴 The early stop rarely fired with shared memory requested Applied. Shared memory is routinely empty for a graph with durable data and includeSharedMemory defaults to true, so the curator answering cleanly empty now settles a plane.
🔴 The 'exit' handler latched nothing Applied. After a worker death every later run hung as well. The failure is latched and future runs reject immediately.
🔴 The authority gate had zero executable coverage Applied. Deleting fromAuthority from both sites failed no test. Two negative-direction tests added (multi-wave, no curator, wave-1 peer proving); verified by mutation that removing either gate fails both. Plus the durable-only stop, the authority-clean-empty stop, and the durable: null skipped-plane round.
🟡 Blank DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS silently disabled retries Applied. Number('') === 0 passed the guard. Blank is now treated as unset; an explicit 0 still disables.
🟡 No CHANGELOG / operator docs for the new knobs Applied. CHANGELOG entry plus a tuning table in docs/use-dkg/backpressure-observability.md.

Round 4 (otReviewAgent follow-ups inside existing threads, 5 🟡) — 3 applied in 687650fda, 2 answered:

Finding Outcome
🟡 Plane-proof rules duplicated inside the worker Applied. The worker hand-coded the proof expression at the early-stop boundary, so a new verified-content signal added to the readiness predicate would have left the walk behind. catchupPeerPlaneEvidence now reduces one peer's round to evidence and both sides go through the same catchupPlaneProvenByData — the walk on one peer's evidence, readiness on the round's sum. Mutation-checked.
🟡 Shared-memory-only fallback lacks worker-level coverage Applied. The scenario itself landed in 216bcbe5a; the missing half was that this branch runs through a different call path (runCatchupPlaneWithPolicy) and so must carry foreground admission itself — now asserted for priority and source on every fallback peer.
🟡 Inline foreground source propagation not asserted Applied. Dropping source while keeping priority would have passed the old assertion while inline catch-up reported as durable:unspecified. The coalescing test now records and asserts both.
🟡 Extract the state machine out of the worker transport Deferred, conceding the strongest part: I kept growing that state across rounds. What changed instead is that every rule now lives in catchup-runner.ts as a pure exported predicate and the worker holds two booleans and a wave cursor — which is the seam the extraction wants in place first. Not in a PR whose diff already alters what that evidence means, on a canary branch.
🟡 Keep the peer-round boundary typed Already applied in def08c3a5, which landed after the comment was written.

Rounds 5–6 (16 → 18 threads; I had been reading un-paginated review results and was seeing 10 of them) — applied in 04f1216da, c141370f2, 4f03a5f1f, 6683d9c7f:

Finding Outcome
🔴 A bootstrap hint could act as the catch-up authority Applied. resolvePreferredSyncPeerId falls back to the authenticated join-approval hint when metadata resolves no curator, and that hint can be stale — peer ids are cryptographic identities, so a curator that rotated its libp2p key leaves an ordinary member on the id it names. Provenance is now an explicit result (classifySyncPeerProvenance, pure, hint captured before resolution) and only 'metadata' may stop the walk.
🔴 An empty curator round settled a PRIVATE plane Applied. Readiness refuses to prove a private plane from an empty response, so stopping on one stranded the walk — skipping fallback peers that may hold authorized private data and turning a recoverable catch-up into unreachable. Emptiness now settles public planes only; a verified private-only response is content and still counts.
🔴 The clean-empty regression test never crossed a wave boundary Applied. catchupWaveSizes(3, 4, 4) is [3], so the test could not observe early stopping at all. Empty peers now fill the whole first wave with the data-bearing peer behind it; mutation now kills it and two siblings, where before it killed none.
🔴 Worker tests stubbed away the main-thread bridge Applied. Every worker test mocked prepareCatchup, so the code that actually derives authoritativePeerId and forwards source was never run. Added a WorkerCatchupRunner agent bridge suite driving real invoke messages; mutation-checked against both named regressions.
🔴 retryDelaysMs was silently ignored rather than erroring Applied. Retained as retryDelaysMs?: never, so a caller passing it via a variable now fails to compile instead of silently getting up to the full budget.
🟡 Authority provenance inferred from a mutable side effect Applied — see the first row; it no longer depends on resolveCuratorPeerId's cache eviction.
🟡 Retry internals on the public agent surface Applied. The backoff curve, env parser and clock seams are off the barrel; in-package tests import them from source.
🟡 Post-worker-exit runs, blank-env parsing, unref(), and a cap test asserting a bound production does not enforce Applied, each mutation-checked.
🟡 Worker result types erased with any Applied in 6683d9c7f — and I had wrongly reported this as done two rounds earlier. See "A correction" below.

Rounds 7–8 (18 → 23 threads) — applied in 8cd3cef79, 7021150e9:

Finding Outcome
🔴 Metadata-only public graphs could not prove an empty catch-up Applied, but not the way suggested. A registered public graph with no Knowledge Assets still serves its own <cg>/_meta definition triples, so its host answers metadata-only, never wire-empty — the whole-round rule could never fire for it and subscribe reported unreachable forever. Accepting any peer's metadata-only round would have resettled #2006 itself (a member holding _meta but no data yet is the commonest state on the network), so it is scoped to the metadata-resolved curator: authorityEmptyPeers, guarded on nobody else having delivered data, public planes only. Symmetric with the rule already in place — if the curator's verified-data round may stand for the whole graph, so may its "I host this and there is nothing in it".
🔴 Bridge tests stubbed the resolver they were meant to prove Applied. The provenance tests now drive resolveCuratorSyncPeer and both lifecycle methods on their real prototypes against a fake agent with a real preferredSyncPeers map — six cases including the eviction side effect. cg-resolve-refresh.test.ts had the same weakness (resolveCuratorPeerId: async () => authoritativePeer, i.e. it asserted the stub) and now runs against real metadata.
🔴 The wait budget did not cover the first queued admission Half applied. The deadline was taken AFTER the first attempt, making the real bound "however long that attempt took, PLUS maxWaitMs" — it is now taken before it. Declined the cancellation half: the only seam reaching an admitted round aborts the whole sync, and #2006's own rejectedTotal: 0 shows those planes were admitted and working, so cancelling at the deadline would have made that run worse. The budget's scope is now stated in code, CHANGELOG and operator docs rather than implying a cap the policy cannot enforce.
🟡 Curator provenance inferred by comparing ids Applied, and it was wrong in the ORDINARY case rather than merely lossy: the join approval normally comes from the curator, so both routes name the same peer and a fully confirmed curator was classified as a bare hint — declining the early stop exactly where it is worth the most. resolveCuratorSyncPeer now returns the provenance of the branch it actually took; classifySyncPeerProvenance is deleted, since the information it needed never reached it.
🟡 Private shared-memory empty settling uncovered Applied. The durable half landed in round 6; the SWM half matters more, since includeSharedMemory defaults on and shared memory is frequently empty on a graph that has durable data. Mutating out the private guard now kills both tests.
🟡 The removed retryDelaysMs contract had no test Applied. catchup-retry-contract.typecheck.ts pins it against the published barrel, and the agent's build script runs test:types, so CI enforces it. Each @ts-expect-error fails the build in both directions — if the option became assignable again, and as an unused suppression if it were deleted outright.
🟡 Extract the walk from the worker body; split the 1.4k-line test file Declined, with reasons and reopen triggers stated on-thread. The shareable decisions are already pure functions shared with the readiness classifier — which is what stops the walk and the verdict drifting; authorityEmptyPeers this round was added in one reducer and one predicate and both picked it up. What remains is a loop whose control flow is the RPC body. The test split needs a harness extraction that would bury this PR's diff, and must carry the env scoping from 8cd3cef79.

Round 9 (25 threads) — applied in fb0db7d31, 19024a1f9:

Finding Outcome
🔴 Integrity rejections could be ignored when proving an empty public plane Applied, with a sharper rule than "another blocking failure". An integrity rejection is not a peer that failed — it is a peer that served content for this graph which then failed verification, so it is positive evidence the graph is not empty. rejectedKcs / dataRejectedMissingMeta now void the verdict ahead of even the curator's own word, where a plain transport or phase failure does not.
🔴 The retryDelaysMs type test would not catch DELETING the member Applied — and my round-8 claim about it was wrong. See "A second correction" below.
🟡 Preferred and authoritative resolution should be one boundary call Applied. prepareCatchup resolved the sync peer twice, once per notion — two <cg>/_meta reads and, for a wallet-address curator, two registry fallbacks per catch-up. Worse, the resolver evicts the bootstrap hint once metadata confirms a curator, so the second call ran against a mutated map and was never the same call. Now one resolveSyncPeerWithProvenance, with authoritativeSyncPeerId as the single definition of which peer may end the walk.
🟡 Walk as an explicit machine; extract the proof model from the runner module; shrink the barrel further Declined on scope, each with reasoning and a reopen trigger on-thread. All three are packaging/structure changes whose diffs would swamp the behavioural one at nine review rounds deep. The proof model is already cohesive and dependency-free, so the extraction stays cheap — and 7021150e9 removed the duplicate cleanPlaneCompletions literal that would otherwise have had to move with it.

Rounds 10–11 (28 threads) — applied in af3c38f0c, 9ac64b5db:

Finding Outcome
🔴 The authority-empty proof was dropped BEFORE readiness was evaluated Applied. cleanCompletionHasResponse gates the denial and no-response branches that run before catchupPlaneReady is ever consulted, and it listed only the three older evidence carriers. So a public graph whose curator answers metadata-only, with any other peer's shared-memory phase refused, returned denied — discarding a durable plane the classifier would have proven ready. Same bug class as #1921's "gate ALL verification consumers", and I walked into it again.
🟡 The kill-switch tested only one of four documented off values Applied. CATCHUP_STOP_ON_PROOF resolves once at module load, so only the spelling a suite happened to set was ever exercised — dropping 'false' would have left an operator running the fan-out they turned off, with every test green. Extracted resolveCatchupStopOnProof; all four spellings, trimming, case-folding, and default-ON for anything unrecognised.
🟡 The removed public constant was not pinned Applied. The type contract covered the removed option but not the removed export, so re-exporting CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS would have regressed silently. Now a @ts-expect-error root import, paired with a positive import of the replacement so the file cannot pass by the surface having decayed.
🔴 retryDelaysMs is a compile-time break on a published 10.0.x Escalated — now resolved; see "Release contract — decided". @origintrail-official/dkg-agent is published publicly (registry currently 10.0.11), so the break is real, not theoretical; in-tree consumers are zero. The reviewer's own confidence note makes this conditional on release policy, so it is escalated rather than resolved unilaterally.
🟡 Walk-as-machine (5th raise); barrel surface (3rd); proof-model extraction (2nd); typed RPC protocol Declined on scope, each with reasoning and a reopen trigger on-thread.

Rounds 12–14 (31 threads) — applied in 753f0dcb9, 1599470e4, 5aacb7d18:

Finding Outcome
🔴 A non-curator with only _meta could still prove a plane empty Applied, and verified against a fresh build before and after rather than reasoned about: member-with-meta + stranger-empty went truefalse, while only strangers and curator hosted-empty stayed true. A peer returning _meta and no data is the ambiguity the round rule cannot resolve — the requester itself logs "peer may have empty or pruned data graph" for it — so combined with an unrelated peer's empty answer it could settle a 40-KA graph as done with zero, which is #2006's own headline symptom. It costs the legitimately-empty graph nothing (the curator settles that through the other proof mode) and is not vacuous — the all-strangers round still proves the plane, pinned by its own test.
🔴 The offline-curator test did not exercise an offline authority Applied. The fixture omitted authoritativePeerId, so the walk took the no-curator branch and neither half of authorityFirst was pinned — I checked both. Fixture fixed; dropping the comparison now fails it. The surviving !== undefined half is labelled defensive in place, because it is genuinely unobservable (catchupWaveSizes(0, …) is [], so a zero-peer walk runs no waves) and I would rather say so than let it read as a clause a test forgot.
🟡 The empty-proof predicate hid two different proof modes Applied. Split into catchupPlaneProvenByAuthorityHostedEmpty and catchupPlaneProvenByUnanimousEmpty, with the shared "content exists, so it is not empty" checks in emptyVerdictContradicted. The split paid for itself immediately: the 🔴 above adds a voider to one branch that provably cannot affect the other.
🟡 The bridge erased the provenance model into a bare string Applied. My own inconsistency — 9ac64b5db introduced authoritativeSyncPeerId as the single definition of "may end the walk", then the bridge restated provenance === 'metadata' inline against provenance?: string. SyncPeerResolution / authoritativeSyncPeerId are now published and consumed as-is. A deliberate exception to the surface-shrinking asked for elsewhere: this is a real cross-package contract, not a test seam.
🟡 Skipped planes as null; collapse the accumulators; type the RPC protocol; extract the proof model; split the test file Declined on scope and filed as #2008, which carries the proposal, why it is cheap now, and the constraint to land it separately.

Round 15 — applied in 96e1c9bbd:

Finding Outcome
🟡 The admission source labels were never verified at their production call sites Applied, and swept as a class. The tests proved a supplied source is normalized and displayed, not that the call sites SUPPLY it — deleting source: 'vm-recovery' left everything green. Of the seven declared sources, three were unpinned: vm-recovery (asserted around, not on), swm-recovery (no coverage), and on-connect (no coverage, and the highest-volume source in production — every reconnect sync flows through that default). Each is now mutation-checked by deleting the label at its call site.
🟡 Walk extraction (6th raise), proof-model module (5th), worker test split (4th) Tracked in #2008.

Rounds 16–17 (34 threads) — applied in 330a9d62e, f58c8273c:

Finding Outcome
🔴 The bridge handoff that produces the authority-ranked peer list was untested Applied. The walk's entire load reduction depends on opening with the curator, but the worker only ever sees an ALREADY-ranked peerIds and every worker test supplied that list itself — so nothing covered "resolve the peer, then rank the live connections against it". New bridge test with out-of-order duplicated connections and a selectCatchupPeers spy. Mutation killed two, one unplanned: dropping the connection de-duplication, which was double-counting a peer with two live connections in connectedPeers.
🟡 The untrusted worker string leaked into the whole sync API Applied. source was typed string everywhere because ONE producer — the Worker RPC — delivers whatever crossed a structured clone. The clamp now happens at that edge (the CLI bridge decodes as unknown and runs normalizeSyncAdmissionSource), and the three agent options carry SyncAdmissionSource. The scheduler still re-clamps: a clamp that cannot be bypassed is worth more than one that merely type-checks. Needed no new exports.
🟡 Two readiness evidence models kept alive Declined on scope, with evidence that strengthens it. I checked my own compat comment's premise and it does not hold: createInlineCatchupRunner's only callers in the tree are two benchmark scripts, and WorkerCatchupRunner always populates cleanPlaneCompletions. The legacy branch is dead in production, so it should be deleted rather than migrated — recorded in #2008 with the search output.
🟡 Route-level coverage of the worker-exit job contract Declined on scope, behaviour verified. The route already does the right thing (catchfailed, finallyfinishedAt, and dedupe only reuses queued/running), so this is coverage depth, not a defect. Unlike the bridge test above, it needs a subscribe-route harness that does not exist. Filed to #2008 with the exact contract to assert.
🟡 Walk-as-policy-abstraction, barrel surface, test split, proof module Tracked in #2008, which now carries seven deferred items from rounds 9–17.

Round 18 — applied in 161332fdc:

Finding Outcome
🔴 Shared-memory metadata could be promoted to hosted-empty proof Applied. "Metadata proves the peer hosts the graph" is a DURABLE fact — <cg>/_meta carries the Context Graph's own definition triples — and I had generalised it across both planes without checking it transfers. It does not: shared memory is a different artifact, and it is contributed by many members rather than owned by the curator, so "the curator has SWM structure but no SWM rows" says nothing about the network. Left generic, a curator's clean SWM round with metadata and no data settled the shared plane and stopped the walk before any member holding the rows was contacted. The reducer now takes an explicit plane discriminator; on the shared plane only a genuine wire-empty response counts. This narrows my own change, not the baseline.

Round 19 — applied in 1ee5c1371:

Finding Outcome
🔴 A curator's empty shared-memory round could falsely settle the public SWM union Applied, and it exposed that 161332fdc had stated this principle while only half-applying it — it stopped shared-memory METADATA counting as hosting evidence but left a wire-empty curator round producing authorityEmptyPeers: 1. Verified against a rebuilt dist before changing anything.

The codebase settles the question explicitly, and load-bearingly rather than aspirationally: planSharedMemorySyncContextGraphs"a PRIVATE CG converges by REPLACE-recovering the current state from its CURATOR (the authoritative SWM replica) … PUBLIC CGs keep the union path"; applyCuratorScope narrows the SWM catch-up peer set to curator peers only when the graph is private; a public CG has "NO authoritative roster"; and the sync responder serves the SWM plane purely from its own local store. Host mode does not rescue it — hosts are arbitrary connected cores keeping a TTL- and byte-capped FIFO, and host catch-up is the documented fallback for when syncing from members returns nothing.

Shared-memory rounds now produce no hosted-empty evidence at all. The plane stays provable by verified DATA from any peer, or as a whole-round verdict once every peer has answered — reachable again precisely because the walk no longer stops.

What this costs, stated plainly. The amplification fix comes from per-plane narrowing, not from the break: once the curator settles durable, every fallback peer takes the durable: null branch, so the 147,246-triple / ~278 MB durable re-pull stays removed. What is given up is the early break on a subscribe whose graph has no shared memory — a handful of cheap wire-empty rounds, in exchange for never skipping a member that holds rows. On an issue titled "reports incomplete graphs as done", that is the right side of the trade.

Round 20 — applied in 3b9e2ffab:

Finding Outcome
🔴 Changing the admission parameters drops priority and cancellation for existing callers Applied as a loud failure, not the suggested compatibility shim. The facts check out — runContextGraphSyncWithBackpressure went from positional (…, priorityOverride?, operationSignal?) to one admission object. TypeScript rejects the old shape, but a JS caller compiled against it would pass a number, destructure to undefined, and silently lose both its priority AND its cancellation — an operation that ignores its abort signal keeps running after the caller gave up. The silence was the defect, and that is what this removes: the old shape now throws, naming the new one, including an AbortSignal passed sixth. No translation shim: the method has no caller outside packages/agent, so a second shape would be carried and tested forever for a caller that does not exist. The general API-break question on a published 10.0.x line remains the release owner's, on the retryDelaysMs thread.

CI. Kosava: adapters + utilities + demo failed a fourth time on adapter-hermes H-AC-31, a 5 s timeout in a package whose only workspace dependency is dkg-core — untouched by this PR, and green on every re-run. Rather than keep re-running it invisibly, filed as #2009 with the occurrence count and a diagnosis (a dynamic await import() inside the test body charged against the 5 s per-test budget on a loaded runner).

Round 21 — applied in 2e8f38b29. Two findings, one class: a changed contract may BREAK a caller, but must never DEGRADE one silently.

Finding Outcome
🔴 retryDelaysMs was still silently ignored at RUNTIME Applied, and the value turned out never to be read at all. Measured against the built dist on an always-deferred run: a verbatim pre-#2006 caller gets 41 attempts / 180,000 ms of blocking where the old ladder gave 3 attempts / 30 ms — a 6000× change in wall time with no diagnostic. It now throws, before the mode branch so a background caller is not exempt. I had argued "silence is the defect" while leaving this one silent; that inconsistency is fixed.
🔴 The round-20 admission guard missed the cancellation-only legacy shape Applied. (…, work, undefined, signal) — the 6th looks absent and defaults to {}, so only the PRESENCE of a 7th argument reveals a caller that still thinks it is passing a cancellation signal. Reproduced against the built dist (returned normally, dropped the signal). A ...legacyPositionalArgs: never[] rest parameter now makes it a compile error AND a runtime throw; the test table went from two legacy shapes to four.

Class sweep, not two instances — this being the third appearance. Everything else the PR reshaped is loud or harmless: the removed root export fails at ESM link time (the dominant path); the injected clock seams are honoured; mode, includeSharedMemory, the plane callbacks, deferredBackpressure and the priority helpers all still drive the same behaviour. One residual is inherent to JavaScript and recorded rather than papered over: reading a removed export off a dynamic import() namespace yields undefined instead of throwing, which is true of every removed export in every package.

Both guards mutation-checked in both directions, including the compile-time half — widening the rest parameter to any[] fails test:types.

Rounds 22–25 — applied in d2c77a69b, cf72f2404, be6da3fca, b649b6775, 01e19e4d5:

Finding Outcome
🔴 A silent CURATOR let a stranger's empty answer prove an empty graph Applied, and the obvious fix was built and REJECTED first. Voiding on failedPeers also kills the verdict when no curator is resolvable at all — the state where the hosted-empty backstop structurally cannot fire — pinning a legitimately empty public graph at unreachable behind one unreachable stranger, and worse on the shared plane which has no backstop by design. Scoped to authorityUnanswered instead: only the curator's silence is decisive.
🔴 An AMBIGUOUS registry match could become a catch-up authority Applied. A trust escalation I introduced: the wallet-registry fallback is documented in the code as arbitrary when several agents share a wallet, which was harmless while it only RANKED the walk. New 'registry' provenance ranks but cannot settle; a unique registration stays authoritative.
🔴 The walk's stop rule ignored round diagnostics the readiness rule consults Applied. I had claimed in a comment that the two "cannot drift apart"; they could. The walk now calls catchupPlaneProvenByAuthorityHostedEmpty itself, with the round's diagnostics, evaluated at the END of each wave so a contradiction from any peer in that wave is visible regardless of arrival order.
🔴 Removed/reshaped options were still silently ignored at RUNTIME Applied, swept as a class after the third instance. retryDelaysMs (measured: 41 attempts / 180,000 ms against the old ladder's 3 / 30 ms), the positional admission arguments including the cancellation-only shape, and — found by my own sweep, unreported by any reviewer — an injected wait with no paired now, which spun 6,873,671 times in a 2-second budget. All now throw.
🟡 plane was optional and defaulted to durable; sources were a duplicated literal set; the changelog lane and the source→scheduler handoff were uncovered; the worker error latch was unpinned All applied, each mutation-checked.
🟡 Structural decomposition (10 items) Tracked in #2008.

Convergence. The following review round produced no new findings — all four items were re-raises of structural work already tracked in #2008. Every 🔴 raised across eighteen rounds is applied or, in the single case of the retryDelaysMs API break, escalated to the release owner with the facts on the thread.

Self-audit while CI ran. Re-proved the older fail-before evidence still holds after this session's refactors, since several of them moved the code those tests target. One near-miss worth recording: hardwiring the kill-switch in packages/agent/**src**/ left catchup-runner-worker-killswitch.test.ts green, which briefly looked like a vacuous test. It is not — CLI tests resolve the agent to packages/agent/**dist**/, so the mutation never reached the loaded code. Mutating the built artifact killed it immediately. Cross-package mutation testing has to edit what the runtime actually loads.

Class sweep after the round-11 bug. Adding an evidence carrier without teaching every consumer is the actual defect, so I swept all nine enumeration sites of verifiedDataPeers / verifiedPrivateOnlyPeers / emptyPeers across cli/src, agent/src and node-ui/src. Each now either handles authorityEmptyPeers or excludes it for a stated reason (catchupPlaneProvenByData — it is not data; the legacy zero-evidence branch — a legacy result carries no authority information). cleanCompletionHasResponse also now takes the shared CatchupPlaneCompletionEvidence type rather than a structural duplicate, so the next carrier cannot be added to the model and omitted from the gate without a type error.

A second correction

In round 8 I said the new retryDelaysMs type test "fails the build in BOTH directions — it errors today if the option were quietly made assignable again, and it errors as an unused suppression if the option were deleted outright." The second half was false. Excess-property checking rejects an object literal against an annotated target whether the member is never or absent entirely, so a literal-only test passes in both worlds. I confirmed it by deleting the member from the built declaration: test:types stayed green — the mutation I should have run before making the claim.

It is now pinned two ways deletion breaks: an indexed access on the member (TS2339), and a stale options variable, where excess properties are permitted and only a declared never can refuse them. Verified in both directions — deletion gives TS2339 plus an unused suppression, re-widening gives TS2322 plus three. The file records why literals cannot carry this contract, so the next person does not re-add one and think it covers deletion.

CI

Bura: cli [3/4] failed on "Daemon did not become ready within 45s" — not a daemon regression: the same test passes locally at HEAD in 12.8 s. This PR's test files set DKG_CATCHUP_* overrides from vi.hoisted on the real process.env, and shard 3 also runs daemon-http-behavior-extra.test.ts, which spawns a real daemon inheriting the parent environment. Scoped with capture/restore in afterAll (8cd3cef79).

A correction

In review round 2 I reported that the plane boundary had been typed, citing def08c3a5. It had not been. git show <sha>:packages/cli/src/catchup-runner-worker-impl.ts | grep -c CatchupDurableResult returns 0 for every commit on this branch before 6683d9c7f. I had made the edit, run a typecheck, seen it pass, and reported it — without confirming it reached the commit. The reviewer's re-raise was correct and its line references were accurate against the real file. It is fixed now, verified against the committed tree rather than the worktree, and I re-audited every other claim I made in review replies the same way — all others are present.

Release contract — decided

CATCHUP_BACKPRESSURE_RETRY_DELAYS_MS and the retryDelaysMs option are removed
from @origintrail-official/dkg-agent, not aliased, and both now fail loudly at
compile time AND runtime. Both described the fixed [100, 250, 500] ladder, which no
longer exists as a mechanism — an alias could only export a schedule the node does not
follow. Silently ignoring the option was measured at 41 retry attempts and 180,000 ms of
blocking against the old ladder's 3 attempts and 30 ms.

Decision taken: ship as-is. Rationale, confirmed with the repo owner:

  • Node operators need do nothing. The node ships as one unit — the CLI depends on the
    agent as workspace:*, so every package moves to the same version on upgrade and no
    node holds a stale caller.
  • The removal is only visible to code outside this repository that installs the agent
    from npm and calls the catch-up retry policy directly — an internal sync-scheduler knob,
    not part of the SDK surface. In-repo consumers: zero.
  • Anything that does hit it gets an immediate error naming the replacement
    (DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS), not a silent behaviour change.

The CHANGELOG Removed entry carries this reasoning and the migration path.

Two findings from that pass are deferred deliberately, both stated here rather than silently:

  • Readiness poisoned under the old rule is not healed. CONTEXT_GRAPH_READINESS_VERSION stays at 1, and the classifier ORs the previously persisted durableVerified back in, so a Context Graph that a pre-fix node marked verified via the old emptyPeers > 0 rule keeps that bit and short-circuits as already-ready. The fix is forward-only. Bumping the version would force one corrective catch-up per subscribed graph on upgrade; that is a broader operational decision than this bug fix should make unilaterally, so it is flagged for the maintainers. An operator can clear a specific graph by re-subscribing.
  • Waves are barriers, not a rolling pool. Each wave is awaited before the next dispatches, so a slow peer stalls its wave rather than being overlapped by a rolling generation, and a present-but-stalling curator sits alone in wave 1. Combined with the larger retry budget this lengthens the worst case for a graph nobody serves. Denial is fast (the requester throws on the sentinel without consuming a timeout), so the cost lands on stalling peers specifically. DKG_CATCHUP_STOP_ON_PROOF=0 restores the rolling single-wave shape.

Test plan

Baseline before any edit: 6 files / 69 tests green on the CLI catch-up suites.
At HEAD 01e19e4d5: 8 files / 144 tests on the CLI catch-up lane and
7 files / 142 tests on the agent sync lane, all green — re-run just now, not
quoted from an earlier round.

  • pnpm --filter @origintrail-official/dkg exec vitest run --config vitest.unit.config.ts test/catchup-runner.test.ts test/catchup-runner-worker-impl.test.ts test/catchup-runner-worker-lifecycle.test.ts test/context-graph-catchup-readiness.test.ts test/context-graph-subscribe-readiness.test.ts test/context-graph-readiness-migration.test.ts test/backpressure-route.test.ts
  • pnpm --filter @origintrail-official/dkg-agent exec vitest run --config vitest.unit.config.ts test/catchup-policy.test.ts test/catchup-concurrency.test.ts test/sync-backpressure.test.ts test/sync-policy.test.ts test/map-with-concurrency.test.ts test/peer-selection.test.ts
  • pnpm --filter @origintrail-official/dkg exec tsc --noEmit and pnpm --filter @origintrail-official/dkg-agent run build (which runs test:types)
  • pnpm build:packages

Fail-before evidence (each new guarantee was proven to be load-bearing)

Guarantee How it was falsified Result without the fix
walk stops on proof re-ran the worker suite with DKG_CATCHUP_STOP_ON_PROOF=0 expected [ 'peer-0', …(19) ] to deeply equal [ 'peer-0' ] — and the plane-narrowing test failed too, i.e. the kill-switch restores the old fan-out faithfully while every other test still passes
empty cannot mask a failure temporarily reverted catchupPlaneProvenByUnanimousEmpty to emptyPeers > 0 12 tests failed, incl. both route-level regression tests (expected 'done' not to be 'done'); file restored and verified byte-identical
worker exit settles pending runs temporarily removed the 'exit' handler body expected 'pending' to be 'rejected'; file restored and verified byte-identical
deadline drives retries injected virtual clock in catchup-policy.test.ts attempts are > 4 (the fixed ladder's exact count) and total sleep never exceeds the budget

Broad lanes — CI is the gate, and it caught a real break

testnet-canary PRs run the full gates, so the sharded CI lanes are the authority here rather than my box.

CI found one genuine regression this PR introduced, which no local lane had surfaced: Tornado: agent [2/10]sync-on-connect-churn"reconciler still retries stale connected peers" asserted trySyncFromPeer's exact argument list, and that helper gained a third argument (the bounded admission origin). Fixed by asserting 'reconcile' — which is the point of the change: reconciler queue pressure must be attributable rather than indistinguishable from sync-on-connect. CI is green on the head commit (all 10 agent shards, the CLI shards, the Kosava lane, EVM integration, SPARQL lint, Knip).

Local lanes, for completeness, with their failures attributed rather than assumed:

  • CLI test:unit → 14 failed files. Re-ran those exact 14 with the working tree checked out at base 36126b52a (full rebuild in between): 13 fail identically at base. The 14th, assertion-cli-smoke.test.ts, passes in isolation at both base and HEAD (4/4, ~45–56 s; it shells out to the CLI four times and was contended in the full-lane run). Environment, not regression — managed-Oxigraph spawn, auto-update, dkg-doctor, notifications-route, status-route-rpc, trust-endpoint-validation, the daemon-wiring suites and the CLI smoke tests.
  • Agent test:unit → 7 failed files. One was the real sync-on-connect-churn break above (now green). The other six are rfc64-* catalog/transport integration suites that fail in isolation on this box with AbortError: The operation was aborted due to timeout after 25–36 s each — and all of them run in the CI agent shards, which are green on this code. Local environment.

An earlier adapter-hermes failure (Test timed out in 5000ms, a profile-backup test in a package whose only workspace dependency is dkg-core) did not recur and was a flake.

Live testnet validation

A disposable edge node was booted from this branch's build against Base testnet (v10.0.11, commit b93a473a, 6–9 connected peers, oxigraph-server backend), and a foreground catch-up was driven through POST /api/context-graph/subscribe.

Queue-origin attribution works on a live nodeGET /api/diagnostics/backpressure, sync-global, mid-catch-up:

lane=durable        queued=3  inflight=0
   queued  op=durable:reconcile          count=2  oldestAgeMs=75207
   queued  op=durable:on-connect         count=1  oldestAgeMs=31820
lane=shared_memory  queued=0  inflight=2
   active  op=shared-memory:catchup-foreground  count=2  oldestAgeMs=68744

Before this PR every one of those rows read just durable / shared-memory, so the fact that the explicit catch-up was holding both inflight slots while reconcile waited 75 s behind it was not visible without reconstructing it from daemon logs. rejectedTotal: 0 throughout — which supports the reading that the 87–109 s waits in the issue were the duration of head-of-line rounds, not queue-depth rejections, and is why the load reduction (not the retry budget) is the primary fix.

The walk itself ran end-to-end on the real network: the job completed with peersTried=9, syncCapablePeers=9, peersNotAttempted=0, peersResponded=5, failedPeers=6 — i.e. no peer proved the plane, so the walk correctly covered the whole peer set instead of stopping early.

That run also surfaced a real regression that the unit tests could not: with no resolvable curator, the single-peer opening wave just prepends a serial round-trip to every round. Fixed in the second commit — the walk now opens at the full concurrency cap unless a sync-capable curator is actually first in the ranked list, which preserves the previous first-round latency while keeping the one-payload behaviour where it is earned.

Not covered live: the multi-peer "one payload instead of six" measurement needs a curator that actually holds the graph plus ≥10 connected peers. That is covered by the unit walk tests and by the kill-switch differential above (20 peers → 1), not by this run.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants