release: promote testnet-canary to main for 10.0.12 - #2072
Conversation
chore: sync main into testnet-canary after 10.0.11
…try-worker fix(agent): autonomously retry durable finalization inbox
…observability feat(observability): unify scheduler backpressure diagnostics
…line test(rfc64): compose current public baseline gate
…tion fix(sync): walk catch-up peers progressively and fail closed on empty rounds (#2006)
feat(sync): W1 measurement contract — source-attributed sync instruments (I1–I9)
feat(w2a): finalized update convergence — canonical contracts + a genuinely per-chain finalized-read lane
fix(chain): select at most two provider origins so shipped RPC pools construct
fix(agent): relieve repeated sync pressure
…tion fix(agent): prevent RS-heal store starvation
| target, | ||
| orderedPeerIds, | ||
| this.vmReconcileRotationNow(), | ||
| resolutionSucceeded, |
There was a problem hiding this comment.
🔴 Bug: Local-curator exact recovery can treat ordinary peers as authoritative absence proof
What's wrong
The new rotation suppression logic uses resolutionSucceeded to decide whether the candidate roster can prove absence. For a local curator, resolution succeeds but yields no remote curator peer after the local peer is filtered out, so the candidate roster can be made only of ordinary connected fallbacks. Their silence should not prove the on-chain KA absent, but this path can back off future recovery as if it did.
Example
Local node is the curator for CG A, but its local VM is missing ordinal 7. resolveCuratorPeerIdsForCg() returns { curatorIsLocal: true, peerIds: [] }; the node has two ordinary connected peers that do not hold the KA. Those peers both return clean-empty exact responses, and the rotation record treats that ordinary roster as confirmed, entering clean-absence backoff instead of continuing fail-open recovery.
Suggested direction
Gate curatorRosterConfirmed on having a non-local authoritative remote curator roster, or otherwise separate local-curator resolution from ordinary fallback peers.
For Agents
In recoverVmReconcileBatch, do not pass resolutionSucceeded through as confirmed proof when curatorResolution.curatorIsLocal left the remote curator roster empty. Preserve ordinary-peer fallback for transport if desired, but mark that roster unconfirmed, or short-circuit local-curator recovery through local materialization. Add a test where local curator plus ordinary connected peers returning clean-absent does not enter clean-absence backoff.
| clearTimeout(this.#timer); | ||
| this.#timer = undefined; | ||
| } | ||
| await this.#inFlight?.catch(() => undefined); |
There was a problem hiding this comment.
🟡 Issue: Finalization recovery worker can block node shutdown indefinitely
What's wrong
The newly introduced autonomous finalization recovery loop has no cancellation or stop deadline. A single hung due-entry replay keeps the worker's in-flight promise pending, and shutdown waits on that promise directly.
Example
If the worker is inside processDueBatch() and an RPC or store operation never resolves, agent.stop() reaches await this.finalizationHandler?.stopRecoveryWorker() and never proceeds to stop libp2p or close stores. Other new drains in the same shutdown path have explicit timeout races, but this one does not.
Suggested direction
Mirror the bounded-drain pattern used for VM reconcile and membership persistence instead of awaiting the worker’s active batch unconditionally.
Confidence note
This assumes processDueBatch can block on chain/store/materialization calls that do not always honor cancellation; the new worker API does not provide an abort signal or bounded stop wait to contain that case.
For Agents
Add a bounded shutdown path for FinalizationRecoveryWorker: thread an AbortSignal into processDueBatch, race the active batch against a shutdown timeout, or explicitly quarantine/retry on next start without waiting forever. Add a lifecycle test where processDueBatch never settles and DKGAgent.stop() still returns or fails with a bounded shutdown error.
| @@ -3532,6 +4034,460 @@ export class SwmHostModeMethods extends DKGAgentBase { | |||
| this.pruneVmReconcileState(); | |||
| } | |||
|
|
|||
| vmReconcileRotationNow(this: DKGAgent): number { | |||
There was a problem hiding this comment.
🟡 Issue: Split exact-recovery rotation out of the SWM host mixin
What's wrong
This PR adds a full process-local state machine directly into an already oversized, multi-purpose agent mixin. The logic is cohesive, but its ownership is not: rotation records, curator rosters, admission cursors, proof capacity, peer selection, network attempts, and lifecycle fences are all interleaved in the host-mode class. That makes future changes require scanning thousands of unrelated SWM/reconcile lines and increases the chance of preserving behavior only by adding more special cases.
Example
Understanding one clean-absence decision now requires following prepareVmReconcileRotationTarget, vmReconcileUncreditedCandidateOrder, settleVmReconcileRotationAttempt, recoverVmReconcileBatch, and state cleanup across the same giant mixin, plus the backing maps added to DKGAgentBase.
Suggested direction
Introduce a small VmExactRecoveryRotation/planner abstraction with methods like prepare, candidateOrder, settle, and clearContextGraph. That deletes most of the new helper surface from the mixin and gives the state transitions a focused home.
For Agents
Extract the exact VM recovery rotation/backoff/proof state into a dedicated module or class owned by VM reconcile. Preserve peer ordering, curator confirmation, clean-absence versus incomplete-cycle semantics, shutdown invalidation, and capacity bounds. Keep recoverVmReconcileBatch as orchestration over that state machine, and move the private-state-heavy tests to the extracted class plus a thin agent wiring test.
| }); | ||
| }, | ||
| storeGraphScopedAsset: async ({ | ||
| storeGraphScopedAsset: ({ |
There was a problem hiding this comment.
🟡 Issue: Extract graph-scoped materialization lifecycle from durable-sync orchestration
What's wrong
The durable-sync lifecycle method has become a hub for lower-level storage, chain-binding, subscription persistence, and shutdown policy. This is more than a long callback: it places feature-specific materialization lifecycle rules in the orchestration layer, which makes the boundary between sync transport and durable graph materialization hard to reason about.
Example
A subscription binding change has to reason about graphScopedStoreClosed, contextGraphBindingGenerations, persistContextGraphSubscriptionStrict, bindSubscriptionOnChainId, shouldQuarantineCommitted, and graphScopedStorePhysicalRuns inside the durable-sync assembly function.
Suggested direction
Move this closure into a dedicated module such as graph-scoped-asset-store or into the existing graph-scoped materialization layer. The lifecycle method should assemble dependencies and call a named boundary, not own every materialization invariant inline.
For Agents
Extract the callback beginning at storeGraphScopedAsset into a focused graph-scoped materialization lifecycle helper. Preserve authentication deadlines, abort behavior, strict binding persistence, quarantine-after-commit behavior, and cache invalidation. Existing durable-sync-lifecycle-binding cases should be able to target the helper directly, with one integration asserting durable sync wires it in.
| // close() fences admission synchronously before the physical-set drain is | ||
| // sampled, so no dispatcher worker can appear behind an observed empty set. | ||
| const dispatcherDrain = vmReconcileDispatcher?.close(); | ||
| const drainPhysicalRuns = async (): Promise<void> => { |
There was a problem hiding this comment.
🟡 Issue: Move subsystem retirement out of the monolithic stop path
What's wrong
The PR concentrates several independent subsystem shutdown protocols in DKGAgent.stop(). That makes the top-level lifecycle method stateful and brittle: adding or modifying any one subsystem requires understanding how all the other drain promises, timeout flags, and restart fences interact.
Example
One stop call now has to know that membership persistence is fenced before callbacks, VM rotation is closed before chain-poller drain, graph-scoped physical stores share the VM reconcile retirement timeout, and membership persistence has its own later timeout/blocked flag.
Suggested direction
Give VM reconcile, graph-scoped materialization, and membership persistence explicit shutdown boundaries that own their own physical run tracking and timeout state. stop() should coordinate those boundaries, not implement each subsystem’s retirement protocol inline.
For Agents
Move the VM reconcile/graph-scoped drain and membership-persistence drain into subsystem-owned closeAndDrain helpers or a small shutdown coordinator. Preserve ordering and timeout behavior, especially the blocked-state retry semantics, but make DKGAgent.stop() read as an ordered list of subsystem shutdown calls.
5 participants
Summary
Promotes the
testnet-canarypayload tomainas the 10.0.12 release candidate.19 commits · 194 files · +32,931 / −1,695.
mainis fully contained intestnet-canary(merge-base ==maintip, 0 commits ahead), so this is a clean promotion with no conflict resolution.No Solidity, ABI, or deployment-registry changes — no on-chain deployment required.
No dashboard schema change —
SCHEMA_VERSIONstays at 31, so no migration and no rollback question.Opened directly from
testnet-canaryrather than a frozen release branch. The frozen-branch pattern existed because canary had no CI coverage; #1992 fixed that, and freezing then only created drift that had to be manually reconciled when a late PR landed (as happened with #1994 on the 10.0.11 cycle).Contents
Plus a series of
test(rfc64)commits hardening the M0 baseline gate, and the #2001 sync back from main.CI status
ci.ymlincludestestnet-canaryin both thepushandpull_requestbranch filters (#1992), so this payload is validated on canary before promotion. At the canary tipc297a7b6f,CI,EVM Integration TestsandKnipall completed success on push. This PR is a confirmation run.Reviewer attention
CHANGELOG.md[Unreleased]is already populated with a single very long### Fixedbullet for Catch-up fan-out overloads sync-global queue and can report incomplete graphs as done #2006. It will need restructuring to house style before the release entry is cut — prior release sections run 11–20 bullets at 3–9k characters total, and this one bullet alone is longer than most complete sections.updated_atinteracts with the TTL prune; worth confirming the self-heal path is intact.Test Plan
maincommit (RELEASE_PROCESS.md §4) — after mergeFollow-up (separate PR, before tagging)
Version bump to 10.0.12 across all 21 manifests, plus the
## [10.0.12]CHANGELOG section built from the existing[Unreleased]content.Not in this release: #1985 (node:sqlite runtime-floor enforcement) and #1971 (curated publish authz, which needs a KAL redeploy) both remain open.
🤖 Generated with Claude Code