fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006) - #2007
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Important
Scope change at
e7f46dca2— the fan-out reduction is NOT in this PR.Ending a catch-up walk early requires knowing which peer is the Context
Graph's curator, and no source available today can establish that: reading the
graph's own
<cg>/_metaidentifies the graph that HOLDS the rows, not thewriter that SUPPLIED them, and ordinary durable-meta catch-up lets any
contacted peer write those very rows. Rather than ship a forgeable early stop
— which would be the same false-
doneclass this issue exists to remove — noresolver route now grants authority, so
authorityProvennever sets, there isno early stop and no per-plane narrowing, and byte volume stays at the
pre-fix level.
This PR therefore ships the correctness and observability half of #2006:
fail-closed readiness, the wall-clock backpressure budget, queue-origin
attribution, and the worker-exit latch. The reported false-
donesymptom isfixed. The amplification (D1) is refiled as #2018, whose first task is the
trusted curator-to-peer binding this needs.
The walk machinery is retained rather than deleted because #2018 re-enables
exactly it; the walk collapses to a single bounded pass while no authority can
resolve, so it costs nothing versus the previous behaviour.
Foreground Context Graph catch-up (the
POST /api/context-graph/subscribejob) pulled the whole graph from every sync-capable peer. On a 14-peer testnet that is 5–13 redundant full payloads — 147,246 fetched triples for a 24,541-triple graph, ~278 MB — which saturates the node-widesync-globalscheduler (2 inflight / 4 queued) and displaces background work. Separately, a clean empty response from an unrelated peer proved a public plane ready, so a run that fetched 122,705 triples and failed five phases settled asdonewith 1 KA out of 40.Progressive peer walk, gated on an authority (present but INERT — see the notice above; no route currently produces an authority, so the walk runs as a single bounded pass). The peer list already arrived ranked authority-first (
orderCatchupPeers: preferred/curator → known cores → rest), but that ordering never became selection. Peers are now contacted in escalating waves (1 → 2 → 4, capped by the existingDKG_CATCHUP_MAX_CONCURRENT_PEERS), fallback peers are narrowed to the planes the curator has not settled, and the walk stops once every requested plane is settled. Only the metadata-resolved curator can settle a plane — any peer'scompleteflag proves only that it served its own manifest — so with no resolvable curator nothing is authority-proven and the walk degrades to the previous full bounded fan-out. The single-peer opening wave is taken only when a sync-capable curator is actually first in the ranked list. The stop decision is evaluated at the END of each wave against the round's accumulated diagnostics, so a contradiction raised by any peer in that wave is visible regardless of arrival order.DKG_CATCHUP_STOP_ON_PROOF=0restores the previous full fan-out.Most of the saving would come from per-plane narrowing, not from the break — but neither is reachable in the shipped build, since narrowing is gated on the same authority. Once the curator settles durable, every fallback peer takes the
durable: nullbranch — which is what removes the 147,246-triple / ~278 MB re-pull — even when the walk keeps going for shared memory.Fail-closed empty proof. An empty response cannot distinguish a peer that hosts an empty graph from one that never heard of it — an unknown CG has no access policy, so the responder authorizes the request and its CG-scoped queries return zero rows. The requester emits
emptyResponsesonly when both phase payloads are empty, so an empty answer can never carry hosting evidence. Readiness is therefore proven by one of three named modes, in order of strength — see the table below.Wall-clock backpressure budget. The foreground retry ladder was a fixed
[100, 250, 500]— 850 ms total — against admitted rounds bounded bySYNC_TOTAL_TIMEOUT_MS(120 s) and measuredsync-globalqueue waits of 87–109 s, so a refused admission always exhausted its budget long before the head of the queue could clear. Replaced with bounded exponential backoff plus jitter against an absolute per-plane deadline (DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS, default 180 s). Waiting costs a timer and no work; the sleep isunref()ed so a pending backoff cannot outliveagent.stop(). Cancellation needs no new plumbing — an aborted admission raises anAbortError, not aSyncBackpressureBusyError, so it never setsdeferredBackpressureand the loop exits on its next check.Queue-origin observability.
sync-globaladmissions now carry a boundedSyncAdmissionSource, so theoperationdimension readsdurable:catchup-foregroundinstead of duplicatinglane. The existing/api/diagnostics/backpressuresnapshot and[backpressure]log records already expose per-operation counts plusoldestQueuedAgeMs/oldestActiveAgeMs, so pressure can now be attributed to a trigger — explicit catch-up vs sync-on-connect vs reconcile — with no core telemetry change. Deliberately not added to the sharedSchedulerPressureTrackerhistograms: that tracker also serves the store scheduler, which passes ~192 rawsourceliterals, so an attribute there would explode cardinality. Unknown origins clamp tounspecifiedat runtime, so a value crossing the worker RPC boundary cannot widen the label space or leak a Context Graph / peer identifier.Worker exit safety (pre-existing, amplified by longer walks).
close()terminates the Worker, which emits'exit'and never'error', so a pendingrun()promise was never settled and the fire-and-forget subscribe job stayedrunningwith nofinishedAt. Because the runner is constructed once per daemon andpostMessageto a dead worker neither throws nor delivers, every later subscribe hung too — and the route's dedupe handed that stuck job back on each retry. The failure is now latched, so pending and future runs both fail fast with a retryable status.Readiness proof model, as it now stands
Twenty-five review rounds moved this materially. This table is the current
behaviour; the chronological round log further down is history, not the model.
catchupPlaneProvenByDatacatchupPlaneProvenByAuthorityHostedEmptycatchupPlaneProvenByUnanimousEmptyFour asymmetries carry the correctness, and each exists because the naive
symmetric rule was shown to be wrong — in every case by a reviewer finding, or
in one case by my own sweep, rather than by design:
Only the curator's emptiness counts, and only on durable.
<cg>/_metacarries the Context Graph's own definition triples, so a curator serving them
proves it hosts the graph. Public shared memory is a per-agent-address layered
union (
<swm>/<addr>/<n>) contributed by many members —planSharedMemorySyncContextGraphsstates it outright: "PUBLIC CGs keep the union path" — so a curator holding
no SWM rows has said nothing about the members' layers.
Content that exists, in any form, beats silence. Verified data, data
fetched anywhere in the round, and content that arrived and failed
verification (
rejectedKcs,dataRejectedMissingMeta) all void an emptyverdict. A non-curator answering
_metawith no data voids it too — that isthe commonest state on the network (a member that has not synced yet) and
accepting it would resettle Catch-up fan-out overloads sync-global queue and can report incomplete graphs as done #2006 itself.
A silent CURATOR voids the verdict; a silent stranger does not.
authorityUnansweredrecords that a resolvable curator was selected and nevercleanly answered. Voiding on
failedPeersinstead would also kill the verdictwhen no curator is resolvable at all — the state where mode 2 structurally
cannot fire — pinning a legitimately empty public graph at
unreachablebehind one unreachable stranger.
Only a DETERMINISTIC curator resolution may end the walk.
<cg>/_metanamesthe curator either as a libp2p peer id directly (legacy) or as a wallet address
(V10). The wallet route prefers the projected
DKG_CREATORtriple, and fallsback to an agent-registry lookup that the code has always documented as
arbitrary when several agents register the same wallet. That was harmless while
this only ranked the walk; it is not, now that it can end one. An ambiguous
match therefore carries
'registry'provenance — it ranks, but only'metadata'is authoritative. A unique registration stays'metadata', sincethat is a deterministic binding.
metadata<cg>/_metadirectly,DKG_CREATOR, or a UNIQUE registry matchregistrybootstrap-hintnoneBoth empty modes are additionally gated, and these are the clauses that close the
reported #2006 round rather than the asymmetries above:
false outright when
isPrivate, since an authorized-but-filtered response isindistinguishable from an empty one on this side of the wire.
failedPhases,timedOutPhases,deniedPhasesanddeferredBackpressure. The reported runhad five failed phases alongside 122,705 fetched triples; either clause kills
it on its own.
authorityUnansweredapply to the whole-round mode only — neither canoverride the curator's own hosted-empty proof, which is voided solely by
evidence that content exists. And
authorityUnansweredkeys on a'metadata'-provenance curator: a'registry'peer is never handed to theworker as an authority, so its silence is not decisive either.
Changed contracts fail loudly, never silently: the removed
retryDelaysMs,the reshaped positional admission arguments, and an injected
waitwithout apaired
nowall throw rather than degrade. Each was measured degrading beforebeing made to fail — the last one spun 6,873,671 times in a 2-second budget.
Deliberate tradeoff
Even the curator's
completeflag proves it served its own manifest, not that the manifest was network-complete (durable-sync.ts:452compares the fetched offset againstmanifestRowCount, derived from the metadata that peer just sent), andSharedMemorySyncResulthas no completion flag at all. Foreground catch-up is therefore optimised for one fast authoritative payload; breadth and eventual convergence remain the background reconcile lane's job, which this PR deliberately leaves fanning out. Gating on the curator bounds the exposure to "the curator's view of its own graph"; the kill-switch exists so even that can be reverted operationally without a redeploy.Scope note: the production foreground runner is
catchup-runner-worker-impl.tsand nothing else —createInlineCatchupRunnerhas no production caller (benchmark scripts only), and every other caller ofsyncContextGraphFromConnectedPeersdefaults tomode: 'background'. The empty-masking defect was CLI-only; the agent-side promotion rule already required inserted data.Related
operationlabel flows straight into feat(observability): add sync pressure and source-cost flame graphs #2005's Grafana flame graph, which already groups by scheduler / lane / operation. No overlap in files.catchup-runner.ts, collapse the parallel per-plane accumulators, and type the worker RPC protocol. Raised across review rounds 9–12 and declined here on scope, not merit; this PR deliberately leaves it cheap (the model is dependency-free, andcleanPlaneCompletionsnow uses the shared evidence type rather than a duplicate).durable-sync.ts:311) andmaterializeVerifiedGraphScopedAssetre-applies an equal version, so every sync re-downloads and re-writes the whole graph. After the fan-out is cut, this is the remaining bulk of the 278 MB.chain.getContextGraphKCCount(cgId)or the locally-unioned<cg>/_metabefore declaring a plane proven would remove the subset tradeoff above.dkg-blackbox-harness): tie per-request status deadlines to the remaining step/campaign deadline and treat transient poll failures as retryable. The issue's harness section is unchanged by this PR.Diagrams
Foreground catch-up peer selection
Before:
sequenceDiagram participant Route as subscribe route participant Worker as catch-up worker participant P0 as peer-0 (curator) participant Pn as peer-1..13 Route->>Worker: run(cg, includeSharedMemory) Worker->>P0: durable + SWM (full graph) Worker->>Pn: durable + SWM (full graph) x13 P0-->>Worker: complete, verified data Pn-->>Worker: 5 x full payload, 5 x failed phase, 5 x clean empty Worker-->>Route: 147k fetched triples, emptyPeers > 0 Route-->>Route: status = done (1 KA of 40)After:
sequenceDiagram participant Route as subscribe route participant Worker as catch-up worker participant P0 as peer-0 (curator) participant Pn as peer-1..13 Route->>Worker: run(cg, includeSharedMemory) Worker->>P0: wave 1 - durable + SWM (P0 is the resolved curator) P0-->>Worker: complete, verified data (both planes) Worker-->>Route: 24.5k fetched triples, peersNotAttempted = 13 Note over Worker,Pn: no curator, or curator did not prove:<br/>waves 2+ walk every peer as before Route-->>Route: status = done (40 KAs)Readiness verdict for an empty response
Before:
sequenceDiagram participant Worker as catch-up worker participant Classifier as readiness classifier participant Job as subscribe job Worker->>Classifier: emptyPeers=5, fetchedData=122705, failedPhases=5 Classifier-->>Classifier: !isPrivate && emptyPeers > 0 Classifier-->>Job: durableVerified = true Job-->>Job: status = doneAfter:
sequenceDiagram participant Worker as catch-up worker participant Classifier as readiness classifier participant Job as subscribe job Worker->>Classifier: emptyPeers=5, fetchedData=122705, failedPhases=5 Classifier-->>Classifier: verified data? no Classifier-->>Classifier: unanimous clean empty? no (content fetched, phases failed) Classifier-->>Job: durableVerified = false Job-->>Job: status = unreachable (retryable)Files changed
32 files, +4,705 / −355 against
origin/testnet-canary. 12 production sourcefiles; 16 test files, 2 vitest lane configs, CHANGELOG and one doc.
packages/cli/src/catchup-runner-worker-impl.tsauthorityUnanswered;peersNotAttemptedpackages/cli/src/catchup-runner.tscatchupPeerPlaneEvidence(plane-discriminated),emptyVerdictContradicted, and the threecatchupPlaneProven*predicates behindcatchupPlaneReady; Worker RPC payloads typed on both ends (the method/arity protocol itself is'exit'/'error'failure latch; clamps the untrusted admission source at the worker edgepackages/cli/src/context-graph-readiness.tscleanCompletionHasResponselists every evidence carrier so a new one cannot be added to the model and omitted from the pre-readiness gatepackages/agent/src/sync/catchup-policy.tsunref()ed sleep; rejects the removedretryDelaysMsand an unpairedwait/nowseampackages/agent/src/sync/catchup-concurrency.tscatchupWaveSizes;resolveCatchupStopOnProof+ theDKG_CATCHUP_STOP_ON_PROOFkill-switchpackages/agent/src/sync/policy.tsSyncAdmissionSourceset + runtimenormalizeSyncAdmissionSourceclamppackages/agent/src/sync/backpressure.tsoperationlabel becomes<work class>:<source>; source threaded into the queue payloadpackages/agent/src/dkg-agent-lifecycle.tsresolveSyncPeerWithProvenance(one resolution, both notions);sourcetyped as the closed union and set at catch-up, on-connect, reconcile, changelog, VM recovery and SWM recovery; rejects the pre-#2006 positional admission shapepackages/agent/src/dkg-agent-cg-resolve.tsresolveCuratorSyncPeerreturns{ peerId, provenance }from the branch it took — comparing ids after resolution could not tell a confirmed curator from an echoed hint. Four-valued provenance: an AMBIGUOUS wallet-registry match ranks but cannot end the walk.authoritativeSyncPeerIdis the single definition of who maypackages/agent/src/index.tspackages/cli/src/api-client.ts,cli-helpers.tspeersNotAttemptedin catch-up status outputCHANGELOG.md,docs/use-dkg/backpressure-observability.md<work class>:<source>label, and the removal's migration path.typecheck.tsfor the removed APIReview rounds
Chronological log of 24 review rounds. The behaviour model above is the current
state — several entries here describe rules that later rounds replaced. Kept for
auditability, including two corrections where I reported something inaccurately.
Round 1 (
otReviewAgent, 3 🔴 / 5 🟡) — 4 applied ind3317b0bf+4c02f87e5, 4 answered in thread:preferredPeerId⇒ all three synced,peersNotAttempted === 0.> 120_000and> 109_000, and a virtual clock keeps retrying past a 90 s capacity clear.CATCHUP_BACKPRESSURE_RETRY_DELAYS_MSbreaks consumerscatchup-runner-worker-killswitch.test.tsboots the worker withDKG_CATCHUP_STOP_ON_PROOF=0and a curator that proves both planes on wave 1, and asserts the full fan-out is restored under the existing concurrency bound.runContextGraphSyncWithBackpressurenow takes a namedadmissionobject; the SWM-recovery call site reads{ source: 'swm-recovery' }instead ofundefined, undefined, 'swm-recovery'.source?: stringsits exactly at the surfaces that can be reconstructed from apostMessagepayload across the Worker RPC, and is clamped once bynormalizeSyncAdmissionSourceinacquire; every layer past the clamp is typedSyncAdmissionSource.4c02f87e5says so at each of the three option surfaces.mapWithConcurrency/CATCHUP_MAX_CONCURRENT_PEER_SYNCS, so the CLI Worker need not deep-importdist/). An internal subpath is a packaging change (exports map, build outputs, publish config) that should be reviewed on its own merits, not as a rider on a sync-path fix.Round 2 (
otReviewAgent, 2 🟡) — both applied indef08c3a5:anyPeerRoundnow carries `CatchupDurableResultrunContextGraphSyncWithBackpressureis now the single normalization point; everything past it carriesSyncAdmissionSource. The clamp inacquirestays as defence in depth and is still pinned by a deliberately bad cast.Round 3 (independent adversarial pass over the final diff, 4 confirmed blockers) — all applied in
216bcbe5a. Three were introduced by this PR:fetchedMetaTriples === 0can essentially never hold for a hosted graph — registration writes definition triples into<cg>/_meta— so a legitimately empty public graph would have become permanentlyunreachable. That clause and thefailedPeersclause are gone; positive tests now pin that neither metadata nor unreachable peers void the verdict.includeSharedMemorydefaults to true, so the curator answering cleanly empty now settles a plane.'exit'handler latched nothingfromAuthorityfrom both sites failed no test. Two negative-direction tests added (multi-wave, no curator, wave-1 peer proving); verified by mutation that removing either gate fails both. Plus the durable-only stop, the authority-clean-empty stop, and thedurable: nullskipped-plane round.DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MSsilently disabled retriesNumber('') === 0passed the guard. Blank is now treated as unset; an explicit0still disables.docs/use-dkg/backpressure-observability.md.Round 4 (
otReviewAgentfollow-ups inside existing threads, 5 🟡) — 3 applied in687650fda, 2 answered:catchupPeerPlaneEvidencenow reduces one peer's round to evidence and both sides go through the samecatchupPlaneProvenByData— the walk on one peer's evidence, readiness on the round's sum. Mutation-checked.216bcbe5a; the missing half was that this branch runs through a different call path (runCatchupPlaneWithPolicy) and so must carry foreground admission itself — now asserted for priority and source on every fallback peer.sourcewhile keepingprioritywould have passed the old assertion while inline catch-up reported asdurable:unspecified. The coalescing test now records and asserts both.catchup-runner.tsas a pure exported predicate and the worker holds two booleans and a wave cursor — which is the seam the extraction wants in place first. Not in a PR whose diff already alters what that evidence means, on a canary branch.def08c3a5, which landed after the comment was written.Rounds 5–6 (16 → 18 threads; I had been reading un-paginated review results and was seeing 10 of them) — applied in
04f1216da,c141370f2,4f03a5f1f,6683d9c7f:resolvePreferredSyncPeerIdfalls back to the authenticated join-approval hint when metadata resolves no curator, and that hint can be stale — peer ids are cryptographic identities, so a curator that rotated its libp2p key leaves an ordinary member on the id it names. Provenance is now an explicit result (classifySyncPeerProvenance, pure, hint captured before resolution) and only'metadata'may stop the walk.unreachable. Emptiness now settles public planes only; a verified private-only response is content and still counts.catchupWaveSizes(3, 4, 4)is[3], so the test could not observe early stopping at all. Empty peers now fill the whole first wave with the data-bearing peer behind it; mutation now kills it and two siblings, where before it killed none.prepareCatchup, so the code that actually derivesauthoritativePeerIdand forwardssourcewas never run. Added aWorkerCatchupRunner agent bridgesuite driving realinvokemessages; mutation-checked against both named regressions.retryDelaysMswas silently ignored rather than erroringretryDelaysMs?: never, so a caller passing it via a variable now fails to compile instead of silently getting up to the full budget.resolveCuratorPeerId's cache eviction.unref(), and a cap test asserting a bound production does not enforceany6683d9c7f— and I had wrongly reported this as done two rounds earlier. See "A correction" below.Rounds 7–8 (18 → 23 threads) — applied in
8cd3cef79,7021150e9:<cg>/_metadefinition triples, so its host answers metadata-only, never wire-empty — the whole-round rule could never fire for it and subscribe reportedunreachableforever. Accepting any peer's metadata-only round would have resettled #2006 itself (a member holding_metabut no data yet is the commonest state on the network), so it is scoped to the metadata-resolved curator:authorityEmptyPeers, guarded on nobody else having delivered data, public planes only. Symmetric with the rule already in place — if the curator's verified-data round may stand for the whole graph, so may its "I host this and there is nothing in it".resolveCuratorSyncPeerand both lifecycle methods on their real prototypes against a fake agent with a realpreferredSyncPeersmap — six cases including the eviction side effect.cg-resolve-refresh.test.tshad the same weakness (resolveCuratorPeerId: async () => authoritativePeer, i.e. it asserted the stub) and now runs against real metadata.maxWaitMs" — it is now taken before it. Declined the cancellation half: the only seam reaching an admitted round aborts the whole sync, and #2006's ownrejectedTotal: 0shows those planes were admitted and working, so cancelling at the deadline would have made that run worse. The budget's scope is now stated in code, CHANGELOG and operator docs rather than implying a cap the policy cannot enforce.resolveCuratorSyncPeernow returns the provenance of the branch it actually took;classifySyncPeerProvenanceis deleted, since the information it needed never reached it.includeSharedMemorydefaults on and shared memory is frequently empty on a graph that has durable data. Mutating out the private guard now kills both tests.retryDelaysMscontract had no testcatchup-retry-contract.typecheck.tspins it against the published barrel, and the agent'sbuildscript runstest:types, so CI enforces it. Each@ts-expect-errorfails the build in both directions — if the option became assignable again, and as an unused suppression if it were deleted outright.authorityEmptyPeersthis round was added in one reducer and one predicate and both picked it up. What remains is a loop whose control flow is the RPC body. The test split needs a harness extraction that would bury this PR's diff, and must carry the env scoping from8cd3cef79.Round 9 (25 threads) — applied in
fb0db7d31,19024a1f9:rejectedKcs/dataRejectedMissingMetanow void the verdict ahead of even the curator's own word, where a plain transport or phase failure does not.retryDelaysMstype test would not catch DELETING the memberprepareCatchupresolved the sync peer twice, once per notion — two<cg>/_metareads and, for a wallet-address curator, two registry fallbacks per catch-up. Worse, the resolver evicts the bootstrap hint once metadata confirms a curator, so the second call ran against a mutated map and was never the same call. Now oneresolveSyncPeerWithProvenance, withauthoritativeSyncPeerIdas the single definition of which peer may end the walk.7021150e9removed the duplicatecleanPlaneCompletionsliteral that would otherwise have had to move with it.Rounds 10–11 (28 threads) — applied in
af3c38f0c,9ac64b5db:cleanCompletionHasResponsegates the denial and no-response branches that run beforecatchupPlaneReadyis ever consulted, and it listed only the three older evidence carriers. So a public graph whose curator answers metadata-only, with any other peer's shared-memory phase refused, returneddenied— discarding a durable plane the classifier would have proven ready. Same bug class as #1921's "gate ALL verification consumers", and I walked into it again.CATCHUP_STOP_ON_PROOFresolves once at module load, so only the spelling a suite happened to set was ever exercised — dropping'false'would have left an operator running the fan-out they turned off, with every test green. ExtractedresolveCatchupStopOnProof; all four spellings, trimming, case-folding, and default-ON for anything unrecognised.CATCHUP_BACKPRESSURE_RETRY_DELAYS_MSwould have regressed silently. Now a@ts-expect-errorroot import, paired with a positive import of the replacement so the file cannot pass by the surface having decayed.retryDelaysMsis a compile-time break on a published10.0.x@origintrail-official/dkg-agentis published publicly (registry currently 10.0.11), so the break is real, not theoretical; in-tree consumers are zero. The reviewer's own confidence note makes this conditional on release policy, so it is escalated rather than resolved unilaterally.Rounds 12–14 (31 threads) — applied in
753f0dcb9,1599470e4,5aacb7d18:_metacould still prove a plane emptymember-with-meta + stranger-emptywenttrue→false, whileonly strangersandcurator hosted-emptystayedtrue. A peer returning_metaand no data is the ambiguity the round rule cannot resolve — the requester itself logs "peer may have empty or pruned data graph" for it — so combined with an unrelated peer's empty answer it could settle a 40-KA graph asdonewith zero, which is #2006's own headline symptom. It costs the legitimately-empty graph nothing (the curator settles that through the other proof mode) and is not vacuous — the all-strangers round still proves the plane, pinned by its own test.authoritativePeerId, so the walk took the no-curator branch and neither half ofauthorityFirstwas pinned — I checked both. Fixture fixed; dropping the comparison now fails it. The surviving!== undefinedhalf is labelled defensive in place, because it is genuinely unobservable (catchupWaveSizes(0, …)is[], so a zero-peer walk runs no waves) and I would rather say so than let it read as a clause a test forgot.catchupPlaneProvenByAuthorityHostedEmptyandcatchupPlaneProvenByUnanimousEmpty, with the shared "content exists, so it is not empty" checks inemptyVerdictContradicted. The split paid for itself immediately: the 🔴 above adds a voider to one branch that provably cannot affect the other.9ac64b5dbintroducedauthoritativeSyncPeerIdas the single definition of "may end the walk", then the bridge restatedprovenance === 'metadata'inline againstprovenance?: string.SyncPeerResolution/authoritativeSyncPeerIdare now published and consumed as-is. A deliberate exception to the surface-shrinking asked for elsewhere: this is a real cross-package contract, not a test seam.null; collapse the accumulators; type the RPC protocol; extract the proof model; split the test fileRound 15 — applied in
96e1c9bbd:source: 'vm-recovery'left everything green. Of the seven declared sources, three were unpinned:vm-recovery(asserted around, not on),swm-recovery(no coverage), andon-connect(no coverage, and the highest-volume source in production — every reconnect sync flows through that default). Each is now mutation-checked by deleting the label at its call site.Rounds 16–17 (34 threads) — applied in
330a9d62e,f58c8273c:peerIdsand every worker test supplied that list itself — so nothing covered "resolve the peer, then rank the live connections against it". New bridge test with out-of-order duplicated connections and aselectCatchupPeersspy. Mutation killed two, one unplanned: dropping the connection de-duplication, which was double-counting a peer with two live connections inconnectedPeers.sourcewas typedstringeverywhere because ONE producer — the Worker RPC — delivers whatever crossed a structured clone. The clamp now happens at that edge (the CLI bridge decodes asunknownand runsnormalizeSyncAdmissionSource), and the three agent options carrySyncAdmissionSource. The scheduler still re-clamps: a clamp that cannot be bypassed is worth more than one that merely type-checks. Needed no new exports.createInlineCatchupRunner's only callers in the tree are two benchmark scripts, andWorkerCatchupRunneralways populatescleanPlaneCompletions. The legacy branch is dead in production, so it should be deleted rather than migrated — recorded in #2008 with the search output.catch→failed,finally→finishedAt, and dedupe only reusesqueued/running), so this is coverage depth, not a defect. Unlike the bridge test above, it needs a subscribe-route harness that does not exist. Filed to #2008 with the exact contract to assert.Round 18 — applied in
161332fdc:<cg>/_metacarries the Context Graph's own definition triples — and I had generalised it across both planes without checking it transfers. It does not: shared memory is a different artifact, and it is contributed by many members rather than owned by the curator, so "the curator has SWM structure but no SWM rows" says nothing about the network. Left generic, a curator's clean SWM round with metadata and no data settled the shared plane and stopped the walk before any member holding the rows was contacted. The reducer now takes an explicitplanediscriminator; on the shared plane only a genuine wire-empty response counts. This narrows my own change, not the baseline.Round 19 — applied in
1ee5c1371:161332fdchad stated this principle while only half-applying it — it stopped shared-memory METADATA counting as hosting evidence but left a wire-empty curator round producingauthorityEmptyPeers: 1. Verified against a rebuiltdistbefore changing anything.The codebase settles the question explicitly, and load-bearingly rather than aspirationally:
planSharedMemorySyncContextGraphs— "a PRIVATE CG converges by REPLACE-recovering the current state from its CURATOR (the authoritative SWM replica) … PUBLIC CGs keep the union path";applyCuratorScopenarrows the SWM catch-up peer set to curator peers only when the graph is private; a public CG has "NO authoritative roster"; and the sync responder serves the SWM plane purely from its own local store. Host mode does not rescue it — hosts are arbitrary connected cores keeping a TTL- and byte-capped FIFO, and host catch-up is the documented fallback for when syncing from members returns nothing.Shared-memory rounds now produce no hosted-empty evidence at all. The plane stays provable by verified DATA from any peer, or as a whole-round verdict once every peer has answered — reachable again precisely because the walk no longer stops.
What this costs, stated plainly. The amplification fix comes from per-plane narrowing, not from the break: once the curator settles durable, every fallback peer takes the
durable: nullbranch, so the 147,246-triple / ~278 MB durable re-pull stays removed. What is given up is the early break on a subscribe whose graph has no shared memory — a handful of cheap wire-empty rounds, in exchange for never skipping a member that holds rows. On an issue titled "reports incomplete graphs as done", that is the right side of the trade.Round 20 — applied in
3b9e2ffab:runContextGraphSyncWithBackpressurewent from positional(…, priorityOverride?, operationSignal?)to oneadmissionobject. TypeScript rejects the old shape, but a JS caller compiled against it would pass a number, destructure toundefined, and silently lose both its priority AND its cancellation — an operation that ignores its abort signal keeps running after the caller gave up. The silence was the defect, and that is what this removes: the old shape now throws, naming the new one, including anAbortSignalpassed sixth. No translation shim: the method has no caller outsidepackages/agent, so a second shape would be carried and tested forever for a caller that does not exist. The general API-break question on a published10.0.xline remains the release owner's, on theretryDelaysMsthread.CI.
Kosava: adapters + utilities + demofailed a fourth time onadapter-hermesH-AC-31, a 5 s timeout in a package whose only workspace dependency isdkg-core— untouched by this PR, and green on every re-run. Rather than keep re-running it invisibly, filed as #2009 with the occurrence count and a diagnosis (a dynamicawait import()inside the test body charged against the 5 s per-test budget on a loaded runner).Round 21 — applied in
2e8f38b29. Two findings, one class: a changed contract may BREAK a caller, but must never DEGRADE one silently.retryDelaysMswas still silently ignored at RUNTIME(…, work, undefined, signal)— the 6th looks absent and defaults to{}, so only the PRESENCE of a 7th argument reveals a caller that still thinks it is passing a cancellation signal. Reproduced against the built dist (returned normally, dropped the signal). A...legacyPositionalArgs: never[]rest parameter now makes it a compile error AND a runtime throw; the test table went from two legacy shapes to four.Class sweep, not two instances — this being the third appearance. Everything else the PR reshaped is loud or harmless: the removed root export fails at ESM link time (the dominant path); the injected clock seams are honoured;
mode,includeSharedMemory, the plane callbacks,deferredBackpressureand the priority helpers all still drive the same behaviour. One residual is inherent to JavaScript and recorded rather than papered over: reading a removed export off a dynamicimport()namespace yieldsundefinedinstead of throwing, which is true of every removed export in every package.Both guards mutation-checked in both directions, including the compile-time half — widening the rest parameter to
any[]failstest:types.Rounds 22–25 — applied in
d2c77a69b,cf72f2404,be6da3fca,b649b6775,01e19e4d5:failedPeersalso kills the verdict when no curator is resolvable at all — the state where the hosted-empty backstop structurally cannot fire — pinning a legitimately empty public graph atunreachablebehind one unreachable stranger, and worse on the shared plane which has no backstop by design. Scoped toauthorityUnansweredinstead: only the curator's silence is decisive.'registry'provenance ranks but cannot settle; a unique registration stays authoritative.catchupPlaneProvenByAuthorityHostedEmptyitself, with the round's diagnostics, evaluated at the END of each wave so a contradiction from any peer in that wave is visible regardless of arrival order.retryDelaysMs(measured: 41 attempts / 180,000 ms against the old ladder's 3 / 30 ms), the positional admission arguments including the cancellation-only shape, and — found by my own sweep, unreported by any reviewer — an injectedwaitwith no pairednow, which spun 6,873,671 times in a 2-second budget. All now throw.planewas optional and defaulted to durable; sources were a duplicated literal set; the changelog lane and the source→scheduler handoff were uncovered; the workererrorlatch was unpinnedConvergence. The following review round produced no new findings — all four items were re-raises of structural work already tracked in #2008. Every 🔴 raised across eighteen rounds is applied or, in the single case of the
retryDelaysMsAPI break, escalated to the release owner with the facts on the thread.Self-audit while CI ran. Re-proved the older fail-before evidence still holds after this session's refactors, since several of them moved the code those tests target. One near-miss worth recording: hardwiring the kill-switch in
packages/agent/**src**/leftcatchup-runner-worker-killswitch.test.tsgreen, which briefly looked like a vacuous test. It is not — CLI tests resolve the agent topackages/agent/**dist**/, so the mutation never reached the loaded code. Mutating the built artifact killed it immediately. Cross-package mutation testing has to edit what the runtime actually loads.Class sweep after the round-11 bug. Adding an evidence carrier without teaching every consumer is the actual defect, so I swept all nine enumeration sites of
verifiedDataPeers/verifiedPrivateOnlyPeers/emptyPeersacrosscli/src,agent/srcandnode-ui/src. Each now either handlesauthorityEmptyPeersor excludes it for a stated reason (catchupPlaneProvenByData— it is not data; the legacy zero-evidence branch — a legacy result carries no authority information).cleanCompletionHasResponsealso now takes the sharedCatchupPlaneCompletionEvidencetype rather than a structural duplicate, so the next carrier cannot be added to the model and omitted from the gate without a type error.A second correction
In round 8 I said the new
retryDelaysMstype test "fails the build in BOTH directions — it errors today if the option were quietly made assignable again, and it errors as an unused suppression if the option were deleted outright." The second half was false. Excess-property checking rejects an object literal against an annotated target whether the member isneveror absent entirely, so a literal-only test passes in both worlds. I confirmed it by deleting the member from the built declaration:test:typesstayed green — the mutation I should have run before making the claim.It is now pinned two ways deletion breaks: an indexed access on the member (
TS2339), and a stale options variable, where excess properties are permitted and only a declarednevercan refuse them. Verified in both directions — deletion givesTS2339plus an unused suppression, re-widening givesTS2322plus three. The file records why literals cannot carry this contract, so the next person does not re-add one and think it covers deletion.CI
Bura: cli [3/4]failed on "Daemon did not become ready within 45s" — not a daemon regression: the same test passes locally at HEAD in 12.8 s. This PR's test files setDKG_CATCHUP_*overrides fromvi.hoistedon the realprocess.env, and shard 3 also runsdaemon-http-behavior-extra.test.ts, which spawns a real daemon inheriting the parent environment. Scoped with capture/restore inafterAll(8cd3cef79).A correction
In review round 2 I reported that the plane boundary had been typed, citing
def08c3a5. It had not been.git show <sha>:packages/cli/src/catchup-runner-worker-impl.ts | grep -c CatchupDurableResultreturns 0 for every commit on this branch before6683d9c7f. I had made the edit, run a typecheck, seen it pass, and reported it — without confirming it reached the commit. The reviewer's re-raise was correct and its line references were accurate against the real file. It is fixed now, verified against the committed tree rather than the worktree, and I re-audited every other claim I made in review replies the same way — all others are present.Release contract — decided
CATCHUP_BACKPRESSURE_RETRY_DELAYS_MSand theretryDelaysMsoption are removedfrom
@origintrail-official/dkg-agent, not aliased, and both now fail loudly atcompile time AND runtime. Both described the fixed
[100, 250, 500]ladder, which nolonger exists as a mechanism — an alias could only export a schedule the node does not
follow. Silently ignoring the option was measured at 41 retry attempts and 180,000 ms of
blocking against the old ladder's 3 attempts and 30 ms.
Decision taken: ship as-is. Rationale, confirmed with the repo owner:
agent as
workspace:*, so every package moves to the same version on upgrade and nonode holds a stale caller.
from npm and calls the catch-up retry policy directly — an internal sync-scheduler knob,
not part of the SDK surface. In-repo consumers: zero.
(
DKG_CATCHUP_BACKPRESSURE_MAX_WAIT_MS), not a silent behaviour change.The CHANGELOG
Removedentry carries this reasoning and the migration path.Two findings from that pass are deferred deliberately, both stated here rather than silently:
CONTEXT_GRAPH_READINESS_VERSIONstays at1, and the classifier ORs the previously persisteddurableVerifiedback in, so a Context Graph that a pre-fix node marked verified via the oldemptyPeers > 0rule keeps that bit and short-circuits as already-ready. The fix is forward-only. Bumping the version would force one corrective catch-up per subscribed graph on upgrade; that is a broader operational decision than this bug fix should make unilaterally, so it is flagged for the maintainers. An operator can clear a specific graph by re-subscribing.DKG_CATCHUP_STOP_ON_PROOF=0restores the rolling single-wave shape.Test plan
Baseline before any edit:
6 files / 69 testsgreen on the CLI catch-up suites.At HEAD
01e19e4d5: 8 files / 144 tests on the CLI catch-up lane and7 files / 142 tests on the agent sync lane, all green — re-run just now, not
quoted from an earlier round.
pnpm --filter @origintrail-official/dkg exec vitest run --config vitest.unit.config.ts test/catchup-runner.test.ts test/catchup-runner-worker-impl.test.ts test/catchup-runner-worker-lifecycle.test.ts test/context-graph-catchup-readiness.test.ts test/context-graph-subscribe-readiness.test.ts test/context-graph-readiness-migration.test.ts test/backpressure-route.test.tspnpm --filter @origintrail-official/dkg-agent exec vitest run --config vitest.unit.config.ts test/catchup-policy.test.ts test/catchup-concurrency.test.ts test/sync-backpressure.test.ts test/sync-policy.test.ts test/map-with-concurrency.test.ts test/peer-selection.test.tspnpm --filter @origintrail-official/dkg exec tsc --noEmitandpnpm --filter @origintrail-official/dkg-agent run build(which runstest:types)pnpm build:packagesFail-before evidence (each new guarantee was proven to be load-bearing)
DKG_CATCHUP_STOP_ON_PROOF=0expected [ 'peer-0', …(19) ] to deeply equal [ 'peer-0' ]— and the plane-narrowing test failed too, i.e. the kill-switch restores the old fan-out faithfully while every other test still passescatchupPlaneProvenByUnanimousEmptytoemptyPeers > 0expected 'done' not to be 'done'); file restored and verified byte-identical'exit'handler bodyexpected 'pending' to be 'rejected'; file restored and verified byte-identicalcatchup-policy.test.ts> 4(the fixed ladder's exact count) and total sleep never exceeds the budgetBroad lanes — CI is the gate, and it caught a real break
testnet-canaryPRs run the full gates, so the sharded CI lanes are the authority here rather than my box.CI found one genuine regression this PR introduced, which no local lane had surfaced:
Tornado: agent [2/10]→sync-on-connect-churn→ "reconciler still retries stale connected peers" assertedtrySyncFromPeer's exact argument list, and that helper gained a third argument (the bounded admission origin). Fixed by asserting'reconcile'— which is the point of the change: reconciler queue pressure must be attributable rather than indistinguishable from sync-on-connect. CI is green on the head commit (all 10 agent shards, the CLI shards, the Kosava lane, EVM integration, SPARQL lint, Knip).Local lanes, for completeness, with their failures attributed rather than assumed:
test:unit→ 14 failed files. Re-ran those exact 14 with the working tree checked out at base36126b52a(full rebuild in between): 13 fail identically at base. The 14th,assertion-cli-smoke.test.ts, passes in isolation at both base and HEAD (4/4, ~45–56 s; it shells out to the CLI four times and was contended in the full-lane run). Environment, not regression — managed-Oxigraph spawn,auto-update,dkg-doctor,notifications-route,status-route-rpc,trust-endpoint-validation, the daemon-wiring suites and the CLI smoke tests.test:unit→ 7 failed files. One was the realsync-on-connect-churnbreak above (now green). The other six arerfc64-*catalog/transport integration suites that fail in isolation on this box withAbortError: The operation was aborted due to timeoutafter 25–36 s each — and all of them run in the CI agent shards, which are green on this code. Local environment.An earlier
adapter-hermesfailure (Test timed out in 5000ms, a profile-backup test in a package whose only workspace dependency isdkg-core) did not recur and was a flake.Live testnet validation
A disposable edge node was booted from this branch's build against Base testnet (
v10.0.11, commitb93a473a, 6–9 connected peers,oxigraph-serverbackend), and a foreground catch-up was driven throughPOST /api/context-graph/subscribe.Queue-origin attribution works on a live node —
GET /api/diagnostics/backpressure,sync-global, mid-catch-up:Before this PR every one of those rows read just
durable/shared-memory, so the fact that the explicit catch-up was holding both inflight slots while reconcile waited 75 s behind it was not visible without reconstructing it from daemon logs.rejectedTotal: 0throughout — which supports the reading that the 87–109 s waits in the issue were the duration of head-of-line rounds, not queue-depth rejections, and is why the load reduction (not the retry budget) is the primary fix.The walk itself ran end-to-end on the real network: the job completed with
peersTried=9,syncCapablePeers=9,peersNotAttempted=0,peersResponded=5,failedPeers=6— i.e. no peer proved the plane, so the walk correctly covered the whole peer set instead of stopping early.That run also surfaced a real regression that the unit tests could not: with no resolvable curator, the single-peer opening wave just prepends a serial round-trip to every round. Fixed in the second commit — the walk now opens at the full concurrency cap unless a sync-capable curator is actually first in the ranked list, which preserves the previous first-round latency while keeping the one-payload behaviour where it is earned.
Not covered live: the multi-peer "one payload instead of six" measurement needs a curator that actually holds the graph plus ≥10 connected peers. That is covered by the unit walk tests and by the kill-switch differential above (20 peers → 1), not by this run.