fix(swm): materialize verified public snapshots on catch-up - #1842
Conversation
…ch-up A node that missed the live gossip never obtained any SWM content. Reproduced deterministically: hold a node out of a publication cycle, restart it, subscribe — 0 of 100 quads after 180s, from five healthy holders. Now 100/100 in ~3s. "0 data + N meta triples" was a red herring: it is CORRECT responder behaviour. Graph-scoped (contentScopeVersion 2) KAs carry no dkg:rootEntity, so the aggregate data phase legitimately returns nothing for them — their content travels as immutable snapshots instead. The catch-up lane fetched and VERIFIED those snapshots and cached them, then never wrote them to the triple store. The held-out node was already holding swm-public-snapshots/81/98/8198388b...nq — the exact 20 quads for KA 27 — on disk, unmaterialized. The asymmetry: live gossip materializes (gossip-publish-handler), durable/VM sync materializes (materializeVerifiedGraphScopedAsset), and PRIVATE CG recovery materializes (swm-recovery.ts materializeReadySnapshot) — but the PUBLIC catch-up lane omitted the step. syncPublicSnapshotsForMeta already exposed an onSnapshotReady hook; the public caller simply never passed it. FIX mirrors the private lane: parse graph-scoped descriptors from verified meta, pass onSnapshotReady, and materialize each verified snapshot via replaceGraph. Deliberate properties: - replaceGraph, NOT insert. A KA graph is all-or-nothing and digest-verified; union-insert risks partial or duplicated graph state across retries, and would bypass per-KA digest verification. - Gated on wsMetaResult.completed. parseGraphScopedSwmRecoveryDescriptors throws on incomplete metadata and this lane pages meta, so an ungated parse would abort the whole context-graph fanout on a timed-out page. - Per-KA error isolation: one unmaterializable snapshot must not take down the rest of the corpus; the phase stays incomplete so the scheduler retries. - storeReplaceGraph is optional on the context so existing callers and test rigs compile unchanged; when absent, materialization is skipped rather than half-applied. TRAP AVOIDED: the tempting fix is to make the data lane work — resurrect dkg:rootEntity, or make readFreshSwmRoots match graph-scoped heads. That reintroduces the O(#KA) aggregate scan the graph-scoped design exists to eliminate and double-transports content. The defect is in materialization, not in the data lane. Because onSnapshotReady fires for 'cache' as well as 'network', nodes that already cached snapshots materialize them on the next pass without refetching. This matches the production symptom on the operator's Base node (fifa CG: query-remote returns 4606 quads while catch-up reports data=0), which may therefore be recoverable locally once this ships. Devnet, clean 6-node, hold-out decisive (baseline 0): before INCOMPLETE — 0/100 after 180788ms after 100/100 quads in 3019ms from remaining holders Full gate 33/35; the two remaining failures are the known redundant chain-reconcile watermark. Agent unit suite 1170 passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ip graphs The materialization fix in d0373b704 replaced KA graphs unconditionally. Live gossip may already hold a RICHER version of the same graph, and replaceGraph is destructive, so catch-up silently DESTROYED content the node already had. Caught end-to-end: a peer that previously converged at 76 quads regressed to 27. Unit tests were green and the build was clean — only the devnet run showed it. The private recovery lane has a guard this port dropped: isGraphAssetMaterialized (an ASK for the head's dkg:assertionGraph marker) skips replacement when the graph is already present. Now wired on both sides, and materialization refuses to run at all when the guard is unavailable rather than proceeding blind. Also corrects the hold-out's decisiveness gate. It required a 0-quad baseline after restart, but that is a PROXY for being held out, and once catch-up works it loses the race: the node can materialize the corpus between restart and the measurement. The real proof is ORDERING — the context graph is created AFTER the node is stopped, so any content it holds must have arrived post-restart, because it did not exist before. Baseline is now INFO; the convergence assertion stands on ordering alone. Verified on a clean 6-node devnet, both directions of the trade-off: #1779 markdown peer population author=78 / peer=76 (was 78/27 while unguarded) hold-out reconstruction 100/100 in 3023ms, baseline 0 full gate 33/35 — the 2 remaining failures are the known redundant chain-reconcile watermark agent unit suite 1170 passing Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| // incomplete metadata, and this lane pages meta, so a timed-out page would | ||
| // otherwise abort the whole CG fanout. A parse failure here must degrade to | ||
| // "no materialization this round" — never take down the sync. | ||
| const snapshotDescriptorsByRef = new Map<string, GraphScopedSwmRecoveryDescriptor[]>(); |
There was a problem hiding this comment.
🟡 Issue: Inline snapshot materialization duplicates the recovery lane instead of reusing it
What's wrong
This adds a second, large implementation of an existing recovery workflow directly into an already busy sync loop. The duplication is structural, not cosmetic: descriptor grouping, idempotence keys, replacement ordering, counters, and error handling are now separate concepts in two files, making the codebase harder to reason about and easier to accidentally diverge.
Example
The same graph-scoped snapshot concept now has two implementations: public sync at this line and private recovery in swm-recovery.ts. A future change to skip logic, metadata handling, or counters has to be made in both places and the two paths can drift.
Suggested direction
Move this orchestration behind a shared GraphScopedSwmSnapshotMaterializer/helper used by both runSharedMemorySync and recoverContextGraphSwm, so the main sync loop stays focused on fetch/verify/store orchestration.
For Agents
Extract the descriptor grouping and ready-snapshot materialization into a shared helper in graph-scoped-swm-recovery.ts or a small requester module. Preserve the current skip-if-materialized behavior, verified snapshot loading, context graph ensure, graph replace, and summary counts; add/adjust focused tests around both callers using the shared helper.
There was a problem hiding this comment.
🟡 Issue: Extract graph-scoped snapshot materialization out of the SWM sync loop
What's wrong
This adds a second, inline graph-scoped materialization flow directly inside an already busy sync orchestrator. The reader now has to track mutable counters, duplicate suppression, lock/version checks, callback timing, and metadata withholding across 100+ lines before understanding when the normal meta/data insert path runs. It also creates two places that must stay aligned as graph-scoped snapshot semantics evolve.
Example
Both packages/agent/src/sync/requester/shared-memory-sync.ts:306 and packages/agent/src/sync/requester/swm-recovery.ts:308 build descriptors by snapshot ref and define a materializeReadySnapshot callback, but with slightly different policy hooks and counters.
Suggested direction
Move this mini state machine behind a dedicated helper or policy object, ideally reusable by the private recovery lane. runSharedMemorySync should orchestrate phases, not also own graph-scoped recovery materialization internals.
For Agents
Look at shared-memory-sync.ts and swm-recovery.ts. Extract a focused graph-scoped snapshot materialization helper that owns descriptor grouping, duplicate suppression, asset materialization, and stats/status reporting while preserving cache/network onSnapshotReady behavior and the new failure-gates-meta behavior.
There was a problem hiding this comment.
🟡 Issue: Extract the graph-scoped snapshot materializer instead of inlining a second recovery lane
What's wrong
This PR adds a second hand-rolled implementation of graph-scoped snapshot materialization in an already busy sync function, while the private recovery lane already owns the same concept. That makes the codebase harder to reason about because future changes to descriptor parsing, metadata completeness, replace ordering, progress accounting, or snapshot readiness now have to be kept aligned across two long paths.
Example
The new public catch-up path builds snapshotDescriptorsByRef, tracks materializedKeys, calls materializeGraphScopedSwmRecoveryAsset, ensures the context graph, replaces the assertion graph, and counts inserted quads. The private recovery path already performs the same core sequence with slightly different policy hooks.
Suggested direction
Move the common parse/group/materialize/replace/progress flow behind one focused helper near graph-scoped-swm-recovery, with small policy hooks for public catch-up vs private recovery. Keep runSharedMemorySync and dkg-agent-lifecycle as orchestration/wiring only.
For Agents
Look in packages/agent/src/sync/graph-scoped-swm-recovery.ts, packages/agent/src/sync/requester/swm-recovery.ts, and packages/agent/src/sync/requester/shared-memory-sync.ts. Extract a shared graph-scoped snapshot materialization helper/service parameterized by the lane-specific policy: public catch-up needs lock/version/content guards and failure gating; private recovery needs verified metadata replacement. Preserve current behavior and run the new materialization test plus existing private SWM recovery tests.
There was a problem hiding this comment.
Partially addressed in 33a621b; the remainder is a reasoned deferral.
What changed: the store-side policy now has one named home — sync/requester/swm-snapshot-materializer.ts (createSharedMemorySnapshotMaterializer) owns the content guard, head read, atomic replace and head-metadata swap, and dkg-agent-lifecycle is reduced to a single factory call wiring agent-owned resources. That removes the biggest drift surface this thread points at (two hand-rolled SPARQL policies living inline in two large files) and makes the policy directly testable (it now is, against a real OxigraphStore).
What is deliberately NOT in this PR: unifying the catch-up loop with swm-recovery.ts's materializeReadySnapshot. The two lanes have different transactional envelopes — public catch-up gates the entire meta insert on materialization success and swaps head metadata ahead of an append-style insert; private recovery inserts per-asset verified meta inline, under a different auth mode and checkpoint namespace. Merging them inside this bugfix would grow its blast radius rather than shrink it. The extracted materializer is the natural seam for that follow-up refactor; proposing to do the lane unification as its own PR on top of it.
There was a problem hiding this comment.
🟡 Issue: Collapse the inlined snapshot-materialization state machine out of runSharedMemorySync
What's wrong
This change makes an already central sync function responsible for storage internals and recovery policy. The new module claims to own materialization, but the caller still orchestrates every important decision, which increases coupling and makes future changes require editing the hot sync path.
Example
The callback runs readStoredHead -> storedVersionOutranksDescriptor -> isGraphAssetMaterialized -> materializeGraphScopedSwmRecoveryAsset -> replaceGraph -> replaceHeadMetadata, then mutates materializationFailures so the later snapshot phase gate can withhold metadata.
Suggested direction
Move the per-descriptor policy behind a single materializer/coordinator method, or extract a syncAndMaterializePublicSnapshots helper shared with the private recovery path. runSharedMemorySync should read as meta/data fetch, verification, snapshot phase, store phase, not a second persistence engine embedded in the middle of the loop.
For Agents
Refactor packages/agent/src/sync/requester/shared-memory-sync.ts and swm-snapshot-materializer.ts so the sync loop delegates one descriptor/snapshot materialization operation and only consumes a compact result. Preserve cache and network onSnapshotReady, superseded-version skip, head repair, materialized quad accounting, and metadata withholding on replace failure.
zsculac
left a comment
There was a problem hiding this comment.
Review completed against base a6f33e4 and head 02e52c8 on both repository-standards and specification-compliance axes.
The implementation does not yet satisfy the stated recovery and no-clobber guarantees. In addition to the existing inline findings about materialization errors being reported as success and omitted subgraph admission context, I added two blocking comments:
- The metadata-marker guard skips the exact pre-upgrade state this PR is meant to repair: cached verified snapshots plus head metadata but no assertion graph.
- The guard and destructive replacement are not serialized with live gossip, so gossip can commit richer content between the check and replacement and still be overwritten.
Verification performed on the pinned head: the runtime package build, agent build, and type checks passed; focused SWM requester/recovery tests passed 37/37; GitHub CI is green. The broader local agent run reached 1,012 passing tests, while 93 network-dependent tests failed because the review sandbox cannot bind local TCP listeners (EPERM), so those failures are environmental and are not attributed to this diff.
Please add targeted regressions for the pre-upgrade metadata-only state, materialization failure retry semantics, admitted subgraphs, and the live-gossip concurrency interleaving.
lupuszr
left a comment
There was a problem hiding this comment.
The happy-path fix is directionally correct, but the current requester can still report a clean catch-up while graph-scoped public assets remain permanently absent. I reproduced both blockers with focused runSharedMemorySync regressions on the exact head: resumed metadata loses earlier descriptors, and a replaceGraph failure is swallowed while its head marker is persisted. These must be fixed before this can prove public-CG convergence.
…content Closes the remaining review blockers on the catch-up materializer. The theme of all of them: the marker could exist without the graph, and the destructive replace could run against state it had not re-verified. LOCK (review P1). materializeReadySnapshot now runs inside the SAME per-KA write lock the live-gossip path takes: swmKaWriteLockKey() is a new shared helper in keyed-lock.ts consumed by BOTH call sites (a hand-rolled copy of the key format would fail silently — an unequal key does not error, it just stops serializing), and the agent passes withKeyedLocks over this.writeLocks — the exact map it already injects into SharedMemoryHandler. The key lowercases the UAL segment: address case varies by source, an under-merged key recreates the race, an over-merged one merely coarsens serialization. VERSION ORDERING (review P1). A lock prevents interleaving, not overwriting-with-older: gossip may advance the KA while catch-up waits. All decisions moved INSIDE the lock, starting with a stored-head assertionVersion read — stored newer than descriptor => skip; unparseable => skip, because state whose ordering we cannot establish must not be destroyed. CONTENT-PROVING GUARD (review P1). isGraphAssetMaterialized now counts the assertion graph and requires exact equality with the descriptor's publicQuadsCount. The prior marker ASK classified the PRE-FIX broken state (head metadata written, graph never written — the observed "0 data + N meta") as materialized, so the repair skipped exactly the nodes that need it, and a partially-fetched metadata round could strand an asset forever behind its own marker. Content-equality also makes multi-round metadata self-healing: a marker without its graph no longer blocks anything. COHESIVE DEPENDENCY (review). The loose optional trio becomes one snapshotMaterializer object — a caller can no longer half-configure materialization silently. TESTS, in CI's include list, driving the REAL runSharedMemorySync with the REAL lock functions: - held-out node materializes a cached snapshot (the production repair path) - the gossip race, deterministically: the test holds the actual lock as "gossip" (the hold IS the pause), commits version 2 while catch-up is provably blocked, releases, and asserts replace never fires — with a checksummed-case UAL on the gossip side so key normalization is exercised - pre-fix broken state heals (marker present, content absent => replaced) - already-materialized asset untouched - failed replace withholds the meta insert and fails the phase Both load-bearing behaviours mutation-tested: disabling the version re-check kills exactly the race test; swallowing failures kills exactly the meta-withholding test. Agent and publisher suites green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
All review findings addressed in 1e9a021. Per-thread disposition: Materialization failure marked successful / permanent (RED) — failures now increment a counter that forces the snapshot phase incomplete, which withholds the meta insert: a head marker can no longer certify a graph that was not written. Regression test asserts a throwing Subgraph-scoped snapshots never materialized (RED) — the parse now receives Multi-round metadata permanently skips earlier assets (RED) + [P1] predicate classifies the pre-fix broken state as materialized — same root cause, one fix: [P1] guard and replace not atomic with live gossip — catch-up now takes the same per-KA lock gossip takes: new shared Data counters (YELLOW) — materialized quads now count into Deferred, deliberately: extracting a shared materializer helper with 🤖 Generated with Claude Code |
… binding Catch-up's meta tail is a union insert outside the lock, so a stale head row can coexist with gossip's newer one. An unordered SELECT taking bindings[0] could then return the older version, defeating the in-lock ordering guard and re-enabling overwrite-with-older on a later pass. Reading the maximum is always the conservative direction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| * distinct KAs into one key would merely coarsen serialization; under-merging | ||
| * recreates the race. Lowercase is therefore the safe direction. | ||
| */ | ||
| export function swmKaWriteLockKey( |
There was a problem hiding this comment.
💡 Suggestion: Keep the SWM lock-key helper in an SWM-owned module
Why it matters
The helper is useful, but placing domain-specific key formatting in the generic lock module and package root makes an internal serialization detail feel like a broad public utility.
Suggestion
Move swmKaWriteLockKey to a module named around workspace/SWM writes, then import/export it from that boundary if cross-package access is required. Leave keyed-lock.ts as the generic concurrency primitive.
There was a problem hiding this comment.
Deferred, with reasoning. Agreed that the package-root export is wider than this helper deserves. Two constraints keep it in keyed-lock.ts for this PR:
- Moving it means editing
workspace-handler.ts's import, and that file is concurrently owned by in-flight work on other PRs this branch must not create conflicts with. - There is a deliberate co-location argument (spelled out in the helper's comment): the key format is load-bearing for lock identity, and drift between two derivations fails silently — an unequal key doesn't error, it just stops serializing. Keeping the derivation next to the primitive it parameterizes is the defensible interim position until an SWM-write boundary module exists to host it.
Proposal: in a follow-up (once workspace-handler.ts is free), move it to a workspace/SWM-write-owned module and re-export from the package boundary for the cross-package consumer in swm-snapshot-materializer.ts.
Adversarial review —
|
|
Review-thread housekeeping — resolved the threads whose findings I could verify are fixed at head Resolved (verified fixed): materialization-failure-marked-successful (failure counter + withheld meta insert, dedicated test), data counters, single Deliberately left OPEN — verified NOT addressed at head:
The open ones are real work, not stale threads. |
…etadata Two review findings on the catch-up materialization path, plus the extraction they both wanted a home for: 1. Count-only materialized-check could skip a newer snapshot. All assertion versions of a graph-scoped KA share ONE graph URI, so an older version with the same quad count read as "already materialized" and the verified newer snapshot was never written, while its metadata could still land — content and head permanently inconsistent. The guard now requires count AND publicQuadsDigest equality: the CONSTRUCT read-back only runs when the count already matches (bounded by exactly the snapshot size we would otherwise write, exact per-KA IRI scope), and the digest-over-roundtrip comparison is the same check resolveWorkspaceOperation already relies on for stored snapshot graphs. 2. Materialization left stale head metadata behind. The sync lane's meta insert is append/union-style, so materializing v2 on top of a v1 head stacked both versions' assertionVersion/shareOperationId rows on one subject — resolveKnowledgeAssetWorkspaceHead reads with LIMIT 1 and could resolve a stale or mixed head. After a successful graph replace (graph FIRST, so a crash never leaves a head without content) the new replaceHeadMetadata deletes the head subject and every operation subject it references — the catch-up counterpart of gossip's delete-then-insert (storeKnowledgeAssetWorkspaceHead) and recovery's replaceMetaForGraphAssets, including its kaUal guard so a corrupt head row can never delete another KA's operation. The fresh verified meta then lands on a clean subject. readStoredHead (MAX-version read, unchanged semantics) now also detects union-insert residue (>1 distinct version/operation) and the skip path collapses it — otherwise a round that failed between replace and head swap would leave the ambiguity permanent, because every later round skips on matching content. Structural: the store-side policy moved out of dkg-agent-lifecycle into createSharedMemorySnapshotMaterializer (swm-snapshot-materializer.ts). The lifecycle now only wires agent-owned resources (store, the SAME lock map SharedMemoryHandler uses, list-cache invalidation); the SPARQL, parsing and replace semantics have a named, directly testable home. Every query in the module is bound to an exact per-KA IRI (head subject / operation subject / assertion graph) — no bucket scans; sparql-scale-lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… subgraph + network paths The regression tests injected isGraphAssetMaterialized, so the actual lifecycle SPARQL implementation was untested — a regression back to a marker-based (or count-only) guard would have stayed green. New swm-snapshot-materializer.test.ts drives the REAL createSharedMemorySnapshotMaterializer against a real OxigraphStore: - marker-without-content (the pre-fix broken state) => guard false - short graph => false; exact content => true (digest survives the store round-trip) - EQUAL-COUNT graph holding another version's content => false — the count-only trap - readStoredHead returns MAX over duplicate head rows and flags the union-insert residue for repair - replaceHeadMetadata deletes head + referenced operations, spares unrelated subjects and other KAs' operations (kaUal guard) - end-to-end: a node fully holding v1 (same quad count as v2) catches up to v2 — graph replaced, exactly ONE head version remains, and the LIMIT-1 production reader resolveKnowledgeAssetWorkspaceHead resolves v2; a second round is a pure no-op (no replace churn) The decision-test file gains the two missing coverage lanes: a KA under a REGISTERED subgraph materializes into its subgraph assertion graph (dropping the parser admission pass-through fails exactly that test), and a cold node fetches the snapshot via the phase='snapshot' network branch and still materializes it / still withholds meta when the replace fails after the fetch. Mutation-tested — each mutant killed by exactly the intended test(s): count-only guard, marker-based guard, MAX->MIN head read, needsRepair=false, head swap removed, skip-path repair removed, subgraph admission removed, network onSnapshotReady dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
packages/evm-module/deployments/localhost_contracts.json is a generated artifact the local build rewrites (branch names, commit hashes, timestamps). None of it is needed by the SWM materialization work; restored byte-identical to origin/main so the diff carries only the actual change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| deps.invalidateListContextGraphsCache(); | ||
| }, | ||
|
|
||
| replaceHeadMetadata: async (contextGraphId, descriptor) => { |
There was a problem hiding this comment.
🟡 Issue: Do not fork the graph-scoped head metadata replacement policy
What's wrong
Two independent implementations of the same metadata replacement rule are now present. That is avoidable architectural drift in a sensitive persistence path, and the new copy is more procedural than the existing query-based shape.
Example
This implementation first selects share IDs from the head, reconstructs urn:dkg:share:${contextGraphId}:${shareId}, then issues one ASK per candidate. The existing recovery path uses a joined SELECT DISTINCT ?op over the head share ID and kaUal. Any future metadata-shape change now has to keep both implementations equivalent.
Suggested direction
Move the head/operation cleanup into one canonical store helper, ideally using the simpler joined query shape and a single deletion loop. The new materializer should depend on that helper rather than carrying a second bespoke cleanup algorithm.
For Agents
Compare packages/agent/src/sync/requester/swm-snapshot-materializer.ts with the lifecycle recovery replaceMetaForGraphAssets context. Extract one shared helper for graph-scoped SWM head metadata replacement and have both private recovery and public catch-up call it. Preserve deletion of the head subject, deletion of only this KA's referenced operation subjects, and preservation of foreign KA operations.
| // Deliberately NOT routed through the sync lane's guarded union insert: | ||
| // a KA graph is all-or-nothing and digest-verified, so it must land via | ||
| // the atomic replace or not at all. | ||
| if (typeof deps.store.replaceGraph !== 'function') { |
There was a problem hiding this comment.
🟡 Issue: Use the canonical atomic graph replacement boundary
What's wrong
This adds another store-capability boundary for atomic graph replacement even though the codebase already has a helper for exactly that contract. The duplicate boundary makes the storage abstraction harder to audit and easier to drift.
Example
Current new path: direct optional-method check followed by deps.store.replaceGraph(...). Existing recovery path: tryReplaceGraphAtomically(dependencies.store, graph, quads, options) and handles clean capability refusal in one place.
Suggested direction
Reuse tryReplaceGraphAtomically instead of open-coding the capability check and direct call. That keeps store capability semantics centralized and makes this materializer match the existing SWM recovery implementation.
For Agents
In swm-snapshot-materializer.ts, replace the direct optional method check with tryReplaceGraphAtomically, or inject a lifecycle-owned replaceGraphAtomically function reused by both recovery paths. Preserve the existing background query options, cache invalidation on success, and failure behavior when atomic replacement is unavailable.
| // the SAME lock map injected into SharedMemoryHandler (sharing | ||
| // the map + key helper is what closes the check-then-replace | ||
| // race with gossip), and list-cache invalidation. | ||
| snapshotMaterializer: createSharedMemorySnapshotMaterializer({ |
There was a problem hiding this comment.
🟡 Issue: Lifecycle materializer wiring is not covered
What's wrong
The core production enablement for this feature is this one wiring point. The added tests cover runSharedMemorySync with injected dependencies and the materializer implementation separately, but not the agent path that connects them, so CI can stay green while production catch-up never materializes snapshots.
Example
Delete or comment out the snapshotMaterializer property in dkg-agent-lifecycle.ts: the new swm-public-snapshot-materialization and swm-snapshot-materializer tests still exercise their hand-built harnesses, but the real agent would go back to caching public snapshots without materializing them during catch-up.
Suggested direction
Add a small integration-style test at the lifecycle boundary so removing or miswiring this property fails.
For Agents
Add a lifecycle-level regression around packages/agent/src/dkg-agent-lifecycle.ts that drives the actual shared-memory sync path, or spies on runSharedMemorySync, and proves LifecycleSyncMethods passes a materializer built with the agent store, the same writeLocks map used by SharedMemoryHandler, and cache invalidation.
The bug
A node that missed the live gossip never obtained any SWM content — permanently.
Reproduced deterministically: hold a node out of a publication cycle, restart it, subscribe.
Root cause
0 data + N meta triplesis a red herring — it is correct responder behaviour. Graph-scoped (contentScopeVersion 2) KAs carry nodkg:rootEntity, so the aggregate data phase legitimately returns nothing for them. Their content travels as immutable snapshots instead.The catch-up lane fetched those snapshots, verified them, cached them — and never wrote them to the triple store. The held-out node was already holding
swm-public-snapshots/81/98/8198388b….nq— the exact 20 quads for KA 27 — on disk, unmaterialized.The asymmetry:
gossip-publish-handler)materializeVerifiedGraphScopedAsset)swm-recovery.ts)syncPublicSnapshotsForMetaalready exposed anonSnapshotReadyhook. The public caller simply never passed it.The fix
Mirrors the private lane: parse graph-scoped descriptors from verified meta, pass
onSnapshotReady, materialize each verified snapshot viareplaceGraph.The first commit replaced graphs unconditionally. Live gossip may already hold a richer version of the same graph, and
replaceGraphis destructive. A peer that previously converged at 76 quads regressed to 27. The build was clean and unit tests were green — only the end-to-end devnet run caught it.The private lane has a guard this port initially dropped:
isGraphAssetMaterialized(an ASK for the head'sdkg:assertionGraphmarker) skips replacement when the graph is already present. Commit 2 wires it on both sides, and materialization now refuses to run at all when the guard is unavailable rather than proceeding blind.Specific reviewer ask: is
isGraphAssetMaterializedthe right sufficiency condition? It is the private lane's own guard, so there is precedent — but it was ported after being bitten, not before.Deliberate properties
replaceGraph, not insert. A KA graph is all-or-nothing and digest-verified; union-insert risks partial/duplicate state across retries and bypasses per-KA digest verification.wsMetaResult.completed.parseGraphScopedSwmRecoveryDescriptorsthrows on incomplete metadata and this lane pages meta, so an ungated parse would abort the whole CG fanout on a timed-out page.Trap avoided
The tempting fix is to make the data lane work — resurrect
dkg:rootEntity, or makereadFreshSwmRootsmatch graph-scoped heads. That reintroduces the O(#KA) aggregate scan the graph-scoped design exists to eliminate, and double-transports content. The defect is in materialization, not the data lane.Possible production link
onSnapshotReadyfires for'cache'as well as'network', so nodes that already cached snapshots materialize them on the next pass without refetching. This matches a live mainnet symptom (a CG returning thousands of quads viaquery-remotewhile catch-up reportsdata=0) — which may therefore be recoverable locally once this ships.Verification
mainbaseReproduce with
test/devnet-public-cg-sync-proof(separate PR):🤖 Generated with Claude Code