Skip to content

fix(swm): materialize verified public snapshots on catch-up - #1842

Merged
branarakic merged 7 commits into
mainfrom
fix/swm-catchup-materialize
Jul 21, 2026
Merged

fix(swm): materialize verified public snapshots on catch-up#1842
branarakic merged 7 commits into
mainfrom
fix/swm-catchup-materialize

Conversation

@branarakic

Copy link
Copy Markdown
Contributor

The bug

A node that missed the live gossip never obtained any SWM content — permanently.

Reproduced deterministically: hold a node out of a publication cycle, restart it, subscribe.

before   INCOMPLETE — 0/100 quads after 180788ms   (from five healthy holders)
after    100/100 quads in 3023ms

Root cause

0 data + N meta triples is a red herring — it is correct responder behaviour. Graph-scoped (contentScopeVersion 2) KAs carry no dkg:rootEntity, so the aggregate data phase legitimately returns nothing for them. Their content travels as immutable snapshots instead.

The catch-up lane fetched those snapshots, verified them, cached them — and never wrote them to the triple store. The held-out node was already holding swm-public-snapshots/81/98/8198388b….nq — the exact 20 quads for KA 27 — on disk, unmaterialized.

The asymmetry:

path materializes?
live gossip (gossip-publish-handler) yes
durable / VM sync (materializeVerifiedGraphScopedAsset) yes
private CG SWM recovery (swm-recovery.ts) yes
public CG SWM catch-up no

syncPublicSnapshotsForMeta already exposed an onSnapshotReady hook. The public caller simply never passed it.

The fix

Mirrors the private lane: parse graph-scoped descriptors from verified meta, pass onSnapshotReady, materialize each verified snapshot via replaceGraph.

⚠️ Please review the guard carefully — commit 2 exists because commit 1 destroyed data

The first commit replaced graphs unconditionally. Live gossip may already hold a richer version of the same graph, and replaceGraph is destructive. A peer that previously converged at 76 quads regressed to 27. The build was clean and unit tests were green — only the end-to-end devnet run caught it.

The private lane has a guard this port initially dropped: isGraphAssetMaterialized (an ASK for the head's dkg:assertionGraph marker) skips replacement when the graph is already present. Commit 2 wires it on both sides, and materialization now refuses to run at all when the guard is unavailable rather than proceeding blind.

Specific reviewer ask: is isGraphAssetMaterialized the right sufficiency condition? It is the private lane's own guard, so there is precedent — but it was ported after being bitten, not before.

Deliberate properties

  • replaceGraph, not insert. A KA graph is all-or-nothing and digest-verified; union-insert risks partial/duplicate state across retries and bypasses per-KA digest verification.
  • Gated on wsMetaResult.completed. parseGraphScopedSwmRecoveryDescriptors throws on incomplete metadata and this lane pages meta, so an ungated parse would abort the whole CG fanout on a timed-out page.
  • Per-KA error isolation. One unmaterializable snapshot must not take down the corpus; the phase stays incomplete so the scheduler retries.
  • Optional deps. Existing callers and test rigs compile unchanged; when absent, materialization is skipped rather than half-applied.

Trap avoided

The tempting fix is to make the data lane work — resurrect dkg:rootEntity, or make readFreshSwmRoots match graph-scoped heads. That reintroduces the O(#KA) aggregate scan the graph-scoped design exists to eliminate, and double-transports content. The defect is in materialization, not the data lane.

Possible production link

onSnapshotReady fires for 'cache' as well as 'network', so nodes that already cached snapshots materialize them on the next pass without refetching. This matches a live mainnet symptom (a CG returning thousands of quads via query-remote while catch-up reports data=0) — which may therefore be recoverable locally once this ships.

Verification

  • clean 6-node devnet, hold-out decisive: 100/100 in 3023ms, baseline 0
  • regression direction also checked: markdown peer population back to author=78 / peer=76 (was 78/27 while unguarded)
  • full gate 33/35 — the two remaining failures are the known redundant chain-reconcile watermark, unrelated
  • agent unit suite 1143 passing on a pure main base

Reproduce with test/devnet-public-cg-sync-proof (separate PR):

RESILIENCE=1 NUM_NODES=6 node devnet/public-cg-sync-proof/proof.mjs

🤖 Generated with Claude Code

Branimir Rakic and others added 2 commits July 20, 2026 09:49
…ch-up

A node that missed the live gossip never obtained any SWM content. Reproduced
deterministically: hold a node out of a publication cycle, restart it, subscribe
— 0 of 100 quads after 180s, from five healthy holders. Now 100/100 in ~3s.

"0 data + N meta triples" was a red herring: it is CORRECT responder behaviour.
Graph-scoped (contentScopeVersion 2) KAs carry no dkg:rootEntity, so the
aggregate data phase legitimately returns nothing for them — their content
travels as immutable snapshots instead. The catch-up lane fetched and VERIFIED
those snapshots and cached them, then never wrote them to the triple store. The
held-out node was already holding
swm-public-snapshots/81/98/8198388b...nq — the exact 20 quads for KA 27 — on
disk, unmaterialized.

The asymmetry: live gossip materializes (gossip-publish-handler), durable/VM sync
materializes (materializeVerifiedGraphScopedAsset), and PRIVATE CG recovery
materializes (swm-recovery.ts materializeReadySnapshot) — but the PUBLIC catch-up
lane omitted the step. syncPublicSnapshotsForMeta already exposed an
onSnapshotReady hook; the public caller simply never passed it.

FIX mirrors the private lane: parse graph-scoped descriptors from verified meta,
pass onSnapshotReady, and materialize each verified snapshot via replaceGraph.

Deliberate properties:
- replaceGraph, NOT insert. A KA graph is all-or-nothing and digest-verified;
  union-insert risks partial or duplicated graph state across retries, and would
  bypass per-KA digest verification.
- Gated on wsMetaResult.completed. parseGraphScopedSwmRecoveryDescriptors throws
  on incomplete metadata and this lane pages meta, so an ungated parse would
  abort the whole context-graph fanout on a timed-out page.
- Per-KA error isolation: one unmaterializable snapshot must not take down the
  rest of the corpus; the phase stays incomplete so the scheduler retries.
- storeReplaceGraph is optional on the context so existing callers and test rigs
  compile unchanged; when absent, materialization is skipped rather than
  half-applied.

TRAP AVOIDED: the tempting fix is to make the data lane work — resurrect
dkg:rootEntity, or make readFreshSwmRoots match graph-scoped heads. That
reintroduces the O(#KA) aggregate scan the graph-scoped design exists to
eliminate and double-transports content. The defect is in materialization, not
in the data lane.

Because onSnapshotReady fires for 'cache' as well as 'network', nodes that
already cached snapshots materialize them on the next pass without refetching.
This matches the production symptom on the operator's Base node (fifa CG:
query-remote returns 4606 quads while catch-up reports data=0), which may
therefore be recoverable locally once this ships.

Devnet, clean 6-node, hold-out decisive (baseline 0):
  before  INCOMPLETE — 0/100 after 180788ms
  after   100/100 quads in 3019ms from remaining holders
Full gate 33/35; the two remaining failures are the known redundant
chain-reconcile watermark. Agent unit suite 1170 passing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ip graphs

The materialization fix in d0373b704 replaced KA graphs unconditionally. Live
gossip may already hold a RICHER version of the same graph, and replaceGraph is
destructive, so catch-up silently DESTROYED content the node already had. Caught
end-to-end: a peer that previously converged at 76 quads regressed to 27. Unit
tests were green and the build was clean — only the devnet run showed it.

The private recovery lane has a guard this port dropped:
isGraphAssetMaterialized (an ASK for the head's dkg:assertionGraph marker) skips
replacement when the graph is already present. Now wired on both sides, and
materialization refuses to run at all when the guard is unavailable rather than
proceeding blind.

Also corrects the hold-out's decisiveness gate. It required a 0-quad baseline
after restart, but that is a PROXY for being held out, and once catch-up works it
loses the race: the node can materialize the corpus between restart and the
measurement. The real proof is ORDERING — the context graph is created AFTER the
node is stopped, so any content it holds must have arrived post-restart, because
it did not exist before. Baseline is now INFO; the convergence assertion stands
on ordering alone.

Verified on a clean 6-node devnet, both directions of the trade-off:
  #1779 markdown peer population  author=78 / peer=76   (was 78/27 while unguarded)
  hold-out reconstruction         100/100 in 3023ms, baseline 0
  full gate                       33/35 — the 2 remaining failures are the known
                                  redundant chain-reconcile watermark
  agent unit suite                1170 passing

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts
// incomplete metadata, and this lane pages meta, so a timed-out page would
// otherwise abort the whole CG fanout. A parse failure here must degrade to
// "no materialization this round" — never take down the sync.
const snapshotDescriptorsByRef = new Map<string, GraphScopedSwmRecoveryDescriptor[]>();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Inline snapshot materialization duplicates the recovery lane instead of reusing it

What's wrong
This adds a second, large implementation of an existing recovery workflow directly into an already busy sync loop. The duplication is structural, not cosmetic: descriptor grouping, idempotence keys, replacement ordering, counters, and error handling are now separate concepts in two files, making the codebase harder to reason about and easier to accidentally diverge.

Example
The same graph-scoped snapshot concept now has two implementations: public sync at this line and private recovery in swm-recovery.ts. A future change to skip logic, metadata handling, or counters has to be made in both places and the two paths can drift.

Suggested direction
Move this orchestration behind a shared GraphScopedSwmSnapshotMaterializer/helper used by both runSharedMemorySync and recoverContextGraphSwm, so the main sync loop stays focused on fetch/verify/store orchestration.

For Agents
Extract the descriptor grouping and ready-snapshot materialization into a shared helper in graph-scoped-swm-recovery.ts or a small requester module. Preserve the current skip-if-materialized behavior, verified snapshot loading, context graph ensure, graph replace, and summary counts; add/adjust focused tests around both callers using the shared helper.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Extract graph-scoped snapshot materialization out of the SWM sync loop

What's wrong
This adds a second, inline graph-scoped materialization flow directly inside an already busy sync orchestrator. The reader now has to track mutable counters, duplicate suppression, lock/version checks, callback timing, and metadata withholding across 100+ lines before understanding when the normal meta/data insert path runs. It also creates two places that must stay aligned as graph-scoped snapshot semantics evolve.

Example
Both packages/agent/src/sync/requester/shared-memory-sync.ts:306 and packages/agent/src/sync/requester/swm-recovery.ts:308 build descriptors by snapshot ref and define a materializeReadySnapshot callback, but with slightly different policy hooks and counters.

Suggested direction
Move this mini state machine behind a dedicated helper or policy object, ideally reusable by the private recovery lane. runSharedMemorySync should orchestrate phases, not also own graph-scoped recovery materialization internals.

For Agents
Look at shared-memory-sync.ts and swm-recovery.ts. Extract a focused graph-scoped snapshot materialization helper that owns descriptor grouping, duplicate suppression, asset materialization, and stats/status reporting while preserving cache/network onSnapshotReady behavior and the new failure-gates-meta behavior.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Extract the graph-scoped snapshot materializer instead of inlining a second recovery lane

What's wrong
This PR adds a second hand-rolled implementation of graph-scoped snapshot materialization in an already busy sync function, while the private recovery lane already owns the same concept. That makes the codebase harder to reason about because future changes to descriptor parsing, metadata completeness, replace ordering, progress accounting, or snapshot readiness now have to be kept aligned across two long paths.

Example
The new public catch-up path builds snapshotDescriptorsByRef, tracks materializedKeys, calls materializeGraphScopedSwmRecoveryAsset, ensures the context graph, replaces the assertion graph, and counts inserted quads. The private recovery path already performs the same core sequence with slightly different policy hooks.

Suggested direction
Move the common parse/group/materialize/replace/progress flow behind one focused helper near graph-scoped-swm-recovery, with small policy hooks for public catch-up vs private recovery. Keep runSharedMemorySync and dkg-agent-lifecycle as orchestration/wiring only.

For Agents
Look in packages/agent/src/sync/graph-scoped-swm-recovery.ts, packages/agent/src/sync/requester/swm-recovery.ts, and packages/agent/src/sync/requester/shared-memory-sync.ts. Extract a shared graph-scoped snapshot materialization helper/service parameterized by the lane-specific policy: public catch-up needs lock/version/content guards and failure gating; private recovery needs verified metadata replacement. Preserve current behavior and run the new materialization test plus existing private SWM recovery tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially addressed in 33a621b; the remainder is a reasoned deferral.

What changed: the store-side policy now has one named home — sync/requester/swm-snapshot-materializer.ts (createSharedMemorySnapshotMaterializer) owns the content guard, head read, atomic replace and head-metadata swap, and dkg-agent-lifecycle is reduced to a single factory call wiring agent-owned resources. That removes the biggest drift surface this thread points at (two hand-rolled SPARQL policies living inline in two large files) and makes the policy directly testable (it now is, against a real OxigraphStore).

What is deliberately NOT in this PR: unifying the catch-up loop with swm-recovery.ts's materializeReadySnapshot. The two lanes have different transactional envelopes — public catch-up gates the entire meta insert on materialization success and swaps head metadata ahead of an append-style insert; private recovery inserts per-asset verified meta inline, under a different auth mode and checkpoint namespace. Merging them inside this bugfix would grow its blast radius rather than shrink it. The extracted materializer is the natural seam for that follow-up refactor; proposing to do the lane unification as its own PR on top of it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Collapse the inlined snapshot-materialization state machine out of runSharedMemorySync

What's wrong
This change makes an already central sync function responsible for storage internals and recovery policy. The new module claims to own materialization, but the caller still orchestrates every important decision, which increases coupling and makes future changes require editing the hot sync path.

Example
The callback runs readStoredHead -> storedVersionOutranksDescriptor -> isGraphAssetMaterialized -> materializeGraphScopedSwmRecoveryAsset -> replaceGraph -> replaceHeadMetadata, then mutates materializationFailures so the later snapshot phase gate can withhold metadata.

Suggested direction
Move the per-descriptor policy behind a single materializer/coordinator method, or extract a syncAndMaterializePublicSnapshots helper shared with the private recovery path. runSharedMemorySync should read as meta/data fetch, verification, snapshot phase, store phase, not a second persistence engine embedded in the middle of the loop.

For Agents
Refactor packages/agent/src/sync/requester/shared-memory-sync.ts and swm-snapshot-materializer.ts so the sync loop delegates one descriptor/snapshot materialization operation and only consumes a compact result. Preserve cache and network onSnapshotReady, superseded-version skip, head repair, materialized quad accounting, and metadata withholding on replace failure.

Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts Outdated
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts Outdated

@zsculac zsculac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against base a6f33e4 and head 02e52c8 on both repository-standards and specification-compliance axes.

The implementation does not yet satisfy the stated recovery and no-clobber guarantees. In addition to the existing inline findings about materialization errors being reported as success and omitted subgraph admission context, I added two blocking comments:

  • The metadata-marker guard skips the exact pre-upgrade state this PR is meant to repair: cached verified snapshots plus head metadata but no assertion graph.
  • The guard and destructive replacement are not serialized with live gossip, so gossip can commit richer content between the check and replacement and still be overwritten.

Verification performed on the pinned head: the runtime package build, agent build, and type checks passed; focused SWM requester/recovery tests passed 37/37; GitHub CI is green. The broader local agent run reached 1,012 passing tests, while 93 network-dependent tests failed because the review sandbox cannot bind local TCP listeners (EPERM), so those failures are environmental and are not attributed to this diff.

Please add targeted regressions for the pre-upgrade metadata-only state, materialization failure retry semantics, admitted subgraphs, and the live-gossip concurrency interleaving.

@lupuszr lupuszr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The happy-path fix is directionally correct, but the current requester can still report a clean catch-up while graph-scoped public assets remain permanently absent. I reproduced both blockers with focused runSharedMemorySync regressions on the exact head: resumed metadata loses earlier descriptors, and a replaceGraph failure is swallowed while its head marker is persisted. These must be fixed before this can prove public-CG convergence.

Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts Outdated
Comment thread packages/agent/src/sync/requester/shared-memory-sync.ts
…content

Closes the remaining review blockers on the catch-up materializer. The theme
of all of them: the marker could exist without the graph, and the destructive
replace could run against state it had not re-verified.

LOCK (review P1). materializeReadySnapshot now runs inside the SAME per-KA
write lock the live-gossip path takes: swmKaWriteLockKey() is a new shared
helper in keyed-lock.ts consumed by BOTH call sites (a hand-rolled copy of the
key format would fail silently — an unequal key does not error, it just stops
serializing), and the agent passes withKeyedLocks over this.writeLocks — the
exact map it already injects into SharedMemoryHandler. The key lowercases the
UAL segment: address case varies by source, an under-merged key recreates the
race, an over-merged one merely coarsens serialization.

VERSION ORDERING (review P1). A lock prevents interleaving, not
overwriting-with-older: gossip may advance the KA while catch-up waits.
All decisions moved INSIDE the lock, starting with a stored-head
assertionVersion read — stored newer than descriptor => skip; unparseable =>
skip, because state whose ordering we cannot establish must not be destroyed.

CONTENT-PROVING GUARD (review P1). isGraphAssetMaterialized now counts the
assertion graph and requires exact equality with the descriptor's
publicQuadsCount. The prior marker ASK classified the PRE-FIX broken state
(head metadata written, graph never written — the observed "0 data + N meta")
as materialized, so the repair skipped exactly the nodes that need it, and a
partially-fetched metadata round could strand an asset forever behind its own
marker. Content-equality also makes multi-round metadata self-healing: a
marker without its graph no longer blocks anything.

COHESIVE DEPENDENCY (review). The loose optional trio becomes one
snapshotMaterializer object — a caller can no longer half-configure
materialization silently.

TESTS, in CI's include list, driving the REAL runSharedMemorySync with the
REAL lock functions:
 - held-out node materializes a cached snapshot (the production repair path)
 - the gossip race, deterministically: the test holds the actual lock as
   "gossip" (the hold IS the pause), commits version 2 while catch-up is
   provably blocked, releases, and asserts replace never fires — with a
   checksummed-case UAL on the gossip side so key normalization is exercised
 - pre-fix broken state heals (marker present, content absent => replaced)
 - already-materialized asset untouched
 - failed replace withholds the meta insert and fails the phase
Both load-bearing behaviours mutation-tested: disabling the version re-check
kills exactly the race test; swallowing failures kills exactly the
meta-withholding test.

Agent and publisher suites green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@branarakic

Copy link
Copy Markdown
Contributor Author

All review findings addressed in 1e9a021. Per-thread disposition:

Materialization failure marked successful / permanent (RED) — failures now increment a counter that forces the snapshot phase incomplete, which withholds the meta insert: a head marker can no longer certify a graph that was not written. Regression test asserts a throwing replaceGraph leaves zero meta batches in the store and failedPhases=1; mutation-tested (swallowing the failure kills exactly that test).

Subgraph-scoped snapshots never materialized (RED) — the parse now receives registeredSubGraphNames/excludedSubGraphNames, same admission as processSharedMemoryBatch.

Multi-round metadata permanently skips earlier assets (RED) + [P1] predicate classifies the pre-fix broken state as materialized — same root cause, one fix: isGraphAssetMaterialized now counts the assertion graph and requires exact equality with publicQuadsCount instead of ASKing for the marker. The pre-fix broken state (marker present, graph never written) reads as not materialized and is repaired — a test seeds exactly that state and asserts the graph gets written. A marker without content no longer blocks any later round, so the checkpoint-resume scenario self-heals on the next full fetch.

[P1] guard and replace not atomic with live gossip — catch-up now takes the same per-KA lock gossip takes: new shared swmKaWriteLockKey() in keyed-lock.ts consumed by both call sites (key drift would fail silently, hence one helper), over the same writeLocks map the agent already injects into SharedMemoryHandler. All decisions moved inside the lock, led by a stored-assertionVersion re-check — stored newer than descriptor means skip; unparseable means skip (never destroy state whose ordering we cannot establish). The deterministic test holds the real lock as "gossip" (the hold is the pause — no sleeps), commits v2 while catch-up is provably blocked, releases, and asserts replace never fires; disabling the version re-check kills exactly that test.

Data counters (YELLOW) — materialized quads now count into insertedDataTriples as well as insertedTriples.
NUL byte (YELLOW) — escaped separator.
Loose optional callbacks (YELLOW) — collapsed into one snapshotMaterializer dependency; half-configuration is no longer expressible.
No regression test (YELLOW)swm-public-snapshot-materialization.test.ts, five cases, added to the vitest include list (this repo's include lists are explicit — a test not listed never runs in CI).

Deferred, deliberately: extracting a shared materializer helper with swm-recovery.ts and de-duplicating the lifecycle ASK closures. Both are real structure improvements, but they reshape the private recovery lane too, and a shared-helper refactor should not ride along with a concurrency fix on a destructive write path. Happy to do them as a follow-up PR.

🤖 Generated with Claude Code

… binding

Catch-up's meta tail is a union insert outside the lock, so a stale head
row can coexist with gossip's newer one. An unordered SELECT taking
bindings[0] could then return the older version, defeating the in-lock
ordering guard and re-enabling overwrite-with-older on a later pass.
Reading the maximum is always the conservative direction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread packages/agent/test/swm-public-snapshot-materialization.test.ts
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/test/swm-public-snapshot-materialization.test.ts Outdated
Comment thread packages/evm-module/deployments/localhost_contracts.json Outdated
* distinct KAs into one key would merely coarsen serialization; under-merging
* recreates the race. Lowercase is therefore the safe direction.
*/
export function swmKaWriteLockKey(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Suggestion: Keep the SWM lock-key helper in an SWM-owned module

Why it matters
The helper is useful, but placing domain-specific key formatting in the generic lock module and package root makes an internal serialization detail feel like a broad public utility.

Suggestion
Move swmKaWriteLockKey to a module named around workspace/SWM writes, then import/export it from that boundary if cross-package access is required. Leave keyed-lock.ts as the generic concurrency primitive.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferred, with reasoning. Agreed that the package-root export is wider than this helper deserves. Two constraints keep it in keyed-lock.ts for this PR:

  1. Moving it means editing workspace-handler.ts's import, and that file is concurrently owned by in-flight work on other PRs this branch must not create conflicts with.
  2. There is a deliberate co-location argument (spelled out in the helper's comment): the key format is load-bearing for lock identity, and drift between two derivations fails silently — an unequal key doesn't error, it just stops serializing. Keeping the derivation next to the primitive it parameterizes is the defensible interim position until an SWM-write boundary module exists to host it.

Proposal: in a follow-up (once workspace-handler.ts is free), move it to a workspace/SWM-write-owned module and re-export from the package boundary for the cross-package consumer in swm-snapshot-materializer.ts.

Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
@Jurij89

Jurij89 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Adversarial review — fix(swm): materialize verified public snapshots on catch-up

Multi-lens review with independent verification against the PR head (02e52c8ca). The core idea (materialize verified snapshots on the public lane, mirroring the private lane) is right, and the destructive-replace guard was the correct instinct. Two findings below, however, are load-bearing: as written the guard defeats the fix for the population you cite as the production motivation, and the parser call silently strands any CG that uses named sub-graphs. Both were confirmed by re-reading the code, not just the diff.


🔴 HIGH — the anti-clobber guard is keyed off a marker this same lane persists unconditionally, so it never repairs an already-stuck node (and permanently strands any KA whose materialize fails once)

isGraphAssetMaterialized is an ASK for the head marker <head> dkg:assertionGraph <graph> (dkg-agent-lifecycle.ts:4778). But that exact marker is written by storeInsert(processed.verifiedMeta) at shared-memory-sync.ts:393-394unconditionally, and after the materialize step at :336. Ordering within one runSharedMemorySync pass:

  1. syncPublicSnapshotsForMeta(...)onSnapshotReadymaterializeReadySnapshot (:336)
  2. storeInsert(processed.verifiedMeta) — writes the <head> dkg:assertionGraph <graph> marker (:393-394)

Two consequences:

  • Already-stuck nodes are not recovered. A node sitting at 0 data + N meta triples already holds that marker from a prior meta-sync pass (the meta insert at :393-394 predates this PR). On the next pass the guard returns truematerializedKeys.add(...); continue; → materialization is skipped. It is never repaired via this lane. That is exactly the population the "Possible production link" section claims becomes recoverable (a CG returning thousands of quads via query-remote while catch-up reports data=0). It does not — those nodes have the marker.
  • A single transient materialize failure strands a KA forever. On a fresh node, if storeReplaceGraph throws for KA X in pass 1, the inner catch swallows it (:174) — but storeInsert(verifiedMeta) at :393-394 still writes X's marker. Pass 2's guard then sees the marker and skips. X stays at zero data permanently.

The held-out devnet repro passes only because a fresh node materializes before the meta insert in the same pass (guard sees no marker yet). The invariant the guard needs — "marker present ⇒ data present" — does not hold on this lane, because the meta phase writes the marker whether or not the data was materialized.

Suggested direction: the guard should reflect data presence, not the meta marker the same lane writes unconditionally — e.g. ASK that the SWM assertion graph is non-empty (ASK { GRAPH <assertionGraph> { ?s ?p ?o } }), or move the head-marker insert to after a successful materialize and make it conditional on it. As-is, "marker exists" and "data was materialized" are independent facts.


🔴 HIGH — the public lane parses descriptors without registeredSubGraphNames, so one sub-graph-scoped SWM head throws and blocks materialization for the entire CG

shared-memory-sync.ts:277 calls the parser with only { contextGraphId, metaQuads }:

for (const descriptor of parseGraphScopedSwmRecoveryDescriptors({
  contextGraphId: pid,
  metaQuads: processed.verifiedMeta,
})) { ... }

The private lane it mirrors passes both (swm-recovery.ts:283): registeredSubGraphNames: recoveryRegistered, excludedSubGraphNames: excluded. With them omitted, allowedWorkspaceMetaGraphs(pid, undefined, undefined) admits only the root .../_shared_memory_meta, so any head living in a named sub-graph meta graph (.../<name>/_shared_memory_meta) hits the hard throw Graph-scoped SWM head … is in an unregistered metadata graph (graph-scoped-swm-recovery.ts:110-112). The parser builds its whole array before returning, so the first such head throws the entire call, the catch at :287-291 runs snapshotDescriptorsByRef.clear(), the onSnapshotReady hook is gated on size > 0 (:349-351) and is never wired — nothing materializes for the CG, root-level KAs included, every round.

Sub-graph heads do reach verifiedMeta: the public lane wires getRegisteredSubGraphNames (dkg-agent-lifecycle.ts:4758) and the verifier retains registered-sub-graph meta. So for any public CG that shares graph-scoped KAs into a named sub-graph (a first-class feature — e.g. per-agent sub-graphs), catch-up materializes nothing and every KA stays at 0 data + N meta. This is not the "per-KA error isolation" the PR describes — the parse step is all-or-nothing per CG.

Fix: forward registeredSubGraphNames/excludedSubGraphNames here exactly as the private lane does (merge getRegisteredSubGraphNames(pid) with discoverSwmRecoverySubGraphNames over processed.verifiedMeta).


🟡 MEDIUM — "the phase stays incomplete so the scheduler retries" does not hold; a per-KA materialize failure is reported as a completed phase

The deliberate-properties section states a failed materialize keeps the phase incomplete for retry. It does not. onSnapshotReady is awaited, but syncPublicSnapshotsForMeta counts the snapshot as readySnapshots and returns completed: true regardless of whether materializeReadySnapshot threw (the inner catch at :174 swallows it). summary.insertedTriples silently excludes the failed KA and no error is surfaced. Retry only happens if a later reconciler-triggered sync re-fires onSnapshotReady('cache') — and per the HIGH finding above, by then the marker exists and the guard skips it. So the practical outcome of a transient failure is permanent, not retried.


🟢 LOW / worth a look

  • Graph-backed snapshots are silently excluded. if (!ref) continue (:282) skips every descriptor whose content rides as an inline publicSnapshotGraph rather than a publicSnapshotRef. Since this lane passes fetchedDataQuads: [], that is arguably correct for the ref-store path — but if any public KA ever uses the inline-graph variant, it is never materialized here and there is no log. Worth an explicit assertion/comment that the public lane only handles the ref variant.
  • Version-blind guard. knowledgeAssetLayerGraphUri derives the SWM graph from agentAddress + kaNumber, not assertionVersion (ka-content-scope.ts:207). Combined with the marker-based guard, a locally-materialized older version can cause catch-up of a newer version to be skipped. Lower confidence — worth confirming whether the head subject is version-stable.
  • Guard vs live-gossip ordering (open question you raised). The recovery lane documents graph-before-marker ordering (swm-recovery.ts:342); the live VM-gossip path writes graph-then-marker under withMaterializationLock and then dropGraph(swmGraph) on VM-finalize (gossip-publish-handler.ts). Whether every SWM producer maintains "marker ⇒ full graph" is what makes isGraphAssetMaterialized sound; the HIGH finding above shows this lane itself breaks it, so it merits a written invariant rather than precedent.

Method: independent multi-lens hunt (correctness / concurrency / guard-soundness / edge-cases) plus two adversarial verifiers per candidate that re-read the head code and tried to refute each; the two HIGH items were additionally re-verified by hand.

@branarakic

Copy link
Copy Markdown
Contributor Author

Review-thread housekeeping — resolved the threads whose findings I could verify are fixed at head f3cfab6c6, each checked against the code (not just the disposition):

Resolved (verified fixed): materialization-failure-marked-successful (failure counter + withheld meta insert, dedicated test), data counters, single snapshotMaterializer dependency, escaped NUL separator, the superseded ASK-copy thread (that call site is now the count predicate; residual dedup explicitly deferred), marker-classifies-broken-state (count-based guard + heal test), guard/replace atomicity (per-KA lock + in-lock version re-check + race test), multi-round skip (marker no longer blocks; count guard self-heals on the next full fetch), swallowed-failure/poisoned-retry.

Deliberately left OPEN — verified NOT addressed at head:

  • Count-only materialization check (🔴, lifecycle:4803) — unchanged; only the MAX version read landed (f3cfab6c6).
  • Stale graph-scoped head metadata after materialization (🔴, in the regression-test thread) — no head/operation-row cleanup exists in the materializer, and resolveKnowledgeAssetWorkspaceHead's LIMIT-1 read is unchanged.
  • Tests stub the guard they verify (🔴, test:146)swm-public-snapshot-materialization.test.ts still injects isGraphAssetMaterialized: async () => overrides.contentPresent?.() ?? false.
  • Subgraph materialization untested (🟡 reply) — the parser fix is in, but none of the 5 tests covers a registered-subgraph KA.
  • Network-fetched snapshot path untested (🟡, test:121) — the harness only drives the cached-snapshot path.
  • localhost_contracts.json churn (🟡) — still 97+/97− vs main in the PR diff.
  • Structural: materializer extraction, lifecycle policy placement, lock-key helper module (deferred/unmoved — left open so the deferral stays visible).

The open ones are real work, not stale threads.

Branimir Rakic and others added 3 commits July 20, 2026 21:43
…etadata

Two review findings on the catch-up materialization path, plus the
extraction they both wanted a home for:

1. Count-only materialized-check could skip a newer snapshot. All
   assertion versions of a graph-scoped KA share ONE graph URI, so an
   older version with the same quad count read as "already
   materialized" and the verified newer snapshot was never written,
   while its metadata could still land — content and head permanently
   inconsistent. The guard now requires count AND publicQuadsDigest
   equality: the CONSTRUCT read-back only runs when the count already
   matches (bounded by exactly the snapshot size we would otherwise
   write, exact per-KA IRI scope), and the digest-over-roundtrip
   comparison is the same check resolveWorkspaceOperation already
   relies on for stored snapshot graphs.

2. Materialization left stale head metadata behind. The sync lane's
   meta insert is append/union-style, so materializing v2 on top of a
   v1 head stacked both versions' assertionVersion/shareOperationId
   rows on one subject — resolveKnowledgeAssetWorkspaceHead reads with
   LIMIT 1 and could resolve a stale or mixed head. After a successful
   graph replace (graph FIRST, so a crash never leaves a head without
   content) the new replaceHeadMetadata deletes the head subject and
   every operation subject it references — the catch-up counterpart of
   gossip's delete-then-insert (storeKnowledgeAssetWorkspaceHead) and
   recovery's replaceMetaForGraphAssets, including its kaUal guard so a
   corrupt head row can never delete another KA's operation. The fresh
   verified meta then lands on a clean subject. readStoredHead
   (MAX-version read, unchanged semantics) now also detects
   union-insert residue (>1 distinct version/operation) and the skip
   path collapses it — otherwise a round that failed between replace
   and head swap would leave the ambiguity permanent, because every
   later round skips on matching content.

Structural: the store-side policy moved out of dkg-agent-lifecycle into
createSharedMemorySnapshotMaterializer (swm-snapshot-materializer.ts).
The lifecycle now only wires agent-owned resources (store, the SAME
lock map SharedMemoryHandler uses, list-cache invalidation); the
SPARQL, parsing and replace semantics have a named, directly testable
home. Every query in the module is bound to an exact per-KA IRI (head
subject / operation subject / assertion graph) — no bucket scans;
sparql-scale-lint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… subgraph + network paths

The regression tests injected isGraphAssetMaterialized, so the actual
lifecycle SPARQL implementation was untested — a regression back to a
marker-based (or count-only) guard would have stayed green. New
swm-snapshot-materializer.test.ts drives the REAL
createSharedMemorySnapshotMaterializer against a real OxigraphStore:

- marker-without-content (the pre-fix broken state) => guard false
- short graph => false; exact content => true (digest survives the
  store round-trip)
- EQUAL-COUNT graph holding another version's content => false — the
  count-only trap
- readStoredHead returns MAX over duplicate head rows and flags the
  union-insert residue for repair
- replaceHeadMetadata deletes head + referenced operations, spares
  unrelated subjects and other KAs' operations (kaUal guard)
- end-to-end: a node fully holding v1 (same quad count as v2) catches
  up to v2 — graph replaced, exactly ONE head version remains, and the
  LIMIT-1 production reader resolveKnowledgeAssetWorkspaceHead resolves
  v2; a second round is a pure no-op (no replace churn)

The decision-test file gains the two missing coverage lanes: a KA under
a REGISTERED subgraph materializes into its subgraph assertion graph
(dropping the parser admission pass-through fails exactly that test),
and a cold node fetches the snapshot via the phase='snapshot' network
branch and still materializes it / still withholds meta when the
replace fails after the fetch.

Mutation-tested — each mutant killed by exactly the intended test(s):
count-only guard, marker-based guard, MAX->MIN head read,
needsRepair=false, head swap removed, skip-path repair removed,
subgraph admission removed, network onSnapshotReady dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
packages/evm-module/deployments/localhost_contracts.json is a generated
artifact the local build rewrites (branch names, commit hashes,
timestamps). None of it is needed by the SWM materialization work;
restored byte-identical to origin/main so the diff carries only the
actual change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
deps.invalidateListContextGraphsCache();
},

replaceHeadMetadata: async (contextGraphId, descriptor) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Do not fork the graph-scoped head metadata replacement policy

What's wrong
Two independent implementations of the same metadata replacement rule are now present. That is avoidable architectural drift in a sensitive persistence path, and the new copy is more procedural than the existing query-based shape.

Example
This implementation first selects share IDs from the head, reconstructs urn:dkg:share:${contextGraphId}:${shareId}, then issues one ASK per candidate. The existing recovery path uses a joined SELECT DISTINCT ?op over the head share ID and kaUal. Any future metadata-shape change now has to keep both implementations equivalent.

Suggested direction
Move the head/operation cleanup into one canonical store helper, ideally using the simpler joined query shape and a single deletion loop. The new materializer should depend on that helper rather than carrying a second bespoke cleanup algorithm.

For Agents
Compare packages/agent/src/sync/requester/swm-snapshot-materializer.ts with the lifecycle recovery replaceMetaForGraphAssets context. Extract one shared helper for graph-scoped SWM head metadata replacement and have both private recovery and public catch-up call it. Preserve deletion of the head subject, deletion of only this KA's referenced operation subjects, and preservation of foreign KA operations.

// Deliberately NOT routed through the sync lane's guarded union insert:
// a KA graph is all-or-nothing and digest-verified, so it must land via
// the atomic replace or not at all.
if (typeof deps.store.replaceGraph !== 'function') {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Use the canonical atomic graph replacement boundary

What's wrong
This adds another store-capability boundary for atomic graph replacement even though the codebase already has a helper for exactly that contract. The duplicate boundary makes the storage abstraction harder to audit and easier to drift.

Example
Current new path: direct optional-method check followed by deps.store.replaceGraph(...). Existing recovery path: tryReplaceGraphAtomically(dependencies.store, graph, quads, options) and handles clean capability refusal in one place.

Suggested direction
Reuse tryReplaceGraphAtomically instead of open-coding the capability check and direct call. That keeps store capability semantics centralized and makes this materializer match the existing SWM recovery implementation.

For Agents
In swm-snapshot-materializer.ts, replace the direct optional method check with tryReplaceGraphAtomically, or inject a lifecycle-owned replaceGraphAtomically function reused by both recovery paths. Preserve the existing background query options, cache invalidation on success, and failure behavior when atomic replacement is unavailable.

// the SAME lock map injected into SharedMemoryHandler (sharing
// the map + key helper is what closes the check-then-replace
// race with gossip), and list-cache invalidation.
snapshotMaterializer: createSharedMemorySnapshotMaterializer({

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Lifecycle materializer wiring is not covered

What's wrong
The core production enablement for this feature is this one wiring point. The added tests cover runSharedMemorySync with injected dependencies and the materializer implementation separately, but not the agent path that connects them, so CI can stay green while production catch-up never materializes snapshots.

Example
Delete or comment out the snapshotMaterializer property in dkg-agent-lifecycle.ts: the new swm-public-snapshot-materialization and swm-snapshot-materializer tests still exercise their hand-built harnesses, but the real agent would go back to caching public snapshots without materializing them during catch-up.

Suggested direction
Add a small integration-style test at the lifecycle boundary so removing or miswiring this property fails.

For Agents
Add a lifecycle-level regression around packages/agent/src/dkg-agent-lifecycle.ts that drives the actual shared-memory sync path, or spies on runSharedMemorySync, and proves LifecycleSyncMethods passes a materializer built with the agent store, the same writeLocks map used by SharedMemoryHandler, and cache invalidation.

@branarakic

Copy link
Copy Markdown
Contributor Author

Canary composition for the fifa-class fixes is up as #1880 (this change carried in full): current testnet-canary + #1868 + #1842, hand-resolved against the #1879 merge, triple-gated green — holdout reconstructs 100/100 quads in ~3s on the exact PR head. Numbers in #1880.

@branarakic
branarakic merged commit d6ca3ae into main Jul 21, 2026
62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants