fix(sync): keep a graph-scoped seal's _meta subject-atomic across durable pages (#1788) by Jurij89 · Pull Request #1916 · OriginTrail/dkg

Jurij89 · 2026-07-22T10:51:22Z

Summary

Durable sync's legacy _meta lane paged the responder's rows with a raw ORDER BY ?g ?s ?p ?o / OFFSET / LIMIT window that can end mid-subject. A graph-scoped author seal is one (g, s) subject with 14 _meta quads; when it straddles a page boundary and a round ends there (the deadline cutoff is between pages), the round delivers only a prefix. Because durable meta is verified per round with no cross-round accumulation, the batch-local control field dkg:assertionVersion is then never admitted — it arrives without its 13 seal siblings, or they arrive without it. On curated CGs this permanently strips the seal to 13/14, and the curator can no longer resolve the author to VM-publish (the exact #1778 symptom, recurring at the sync layer).

The fix makes the durable-meta responder emit pages that always END on a (g, s) subject boundary: a subject straddling the requested row limit is emitted in full — the page EXTENDS past limit, never trims. Each sync round ends on a page boundary and every page now ends on a subject boundary, so each round's accumulated meta ends on a subject boundary, the checkpoint offset is subject-aligned, and a seal's 14 rows are never split across a round. This mirrors the changelog lane's per-graph atomicity without coupling seal admission to a receiver-side store re-read.

Why EXTEND, not TRIM: trimming a page to a subject boundary could empty it when a single subject fills the whole window, and an empty page is the requester's EOF signal — trimming would risk exactly the data-loss class we are fixing. Extending never empties or shortens a page. It is safe because durable meta uses byte-budget pagination (the requester's page size is 8192 > the 500 legacy cap), so the requester never treats an over-sized page as EOF; it advances by the actual row count and the next OFFSET lands on the next subject.

Bounds / safety: _meta subjects are small (a seal is 14 quads; KA descriptors ~10; membership/activity rows bounded), so the extension is O(one subject) and negligible against the #1868 64k-row meta snapshot ceiling. A pathologically large subject is emitted whole rather than truncated (truncation is precisely the #1788 defect), with the transport frame limit as the final guard — a loud failure instead of silent seal corruption.

Scope: the assetUals VM-recovery meta path is intentionally untouched — the seal subject never appears in its VALUES ?s list. No wire-format, schema, or persisted-state change; fully backward/forward compatible (pages are still valid N-Quads; only their boundaries move).

Fixes Durable sync splits a graph-scoped assertion seal's _meta across batches, permanently stripping dkg:assertionVersion (13/14) on curated CGs #1788
Blocks the end-to-end scenario of bug: rootless named-KA SWM gossip drops assertion seal, blocking curator VM publish #1778 / PR fix(publish): let a curator VM-publish a member-shared rootless KA (#1778) #1780 (curator VM-publishes a member-shared rootless KA) on curated CGs under sync contention
Related to fix(sync): serve TTL-filtered SWM meta past the 64,000-row snapshot ceiling #1868 (SWM meta 64k-row snapshot ceiling — the per-subject extension is negligible against it)

Files changed

File	What
`packages/agent/src/sync/responder/graph-plan.ts`	`metaSubjectKey` helper; `subjectAtomic` extend on the cached (snapshot) lane; `readDurableMetaRowsPageSubjectAtomic` + complete-trailing-subject re-read for the store-paged (no-session / oversized-fallback) lane; wired both into `readDurableMetaPage`
`packages/agent/test/sync-responder-durable-meta-subject-atomic.test.ts`	New regression: a seal straddling a page boundary materializes all 14 quads incl. `dkg:assertionVersion` (0-or-14 per page, no dup/gap) on both loaders, plus a clean-boundary no-over-extend case
`packages/agent/test/sync-responder-snapshot-cache.test.ts`	Relax the anti-copy guard to a bounded one-subject boundary peek (hard-fail preserved on any read before offset and on whole-snapshot iteration)
`packages/agent/test/sync-responder-concurrent-interleaving.test.ts`	Rework two cases whose single-subject data encoded the now-fixed cross-page split to use distinct subjects; document the `limit + 1` lookahead
`packages/agent/vitest.unit.config.ts`	Register the new regression test in the unit include list

Test plan

pnpm exec tsc --noEmit (agent package) — clean
node scripts/sparql-scale-lint.mjs --diff <testnet-canary> HEAD — 0 new blocking (the new query uses a bound subject, so no scan finding)
Regression fails-before / passes-after: with the fix disabled, both straddle cases fail split 5/14 (exact Durable sync splits a graph-scoped assertion seal's _meta across batches, permanently stripping dkg:assertionVersion (13/14) on curated CGs #1788), the clean-boundary case still passes; restored → all pass
pnpm exec vitest run --config vitest.unit.config.ts across 20 durable/sync responder suites — 262 tests pass
- sync-responder-durable-meta-subject-atomic, sync-responder-snapshot-cache, sync-responder-oversized-fallback, sync-responder-cursor, sync-responder-concurrent-interleaving, sync-byte-budget-pages, sync-responder-swm-meta-ceiling, durable-meta-admission, sync-control-metadata-admission, durable-integrity-seal-assertion-version, and more

🤖 Generated with Claude Code

…able pages (#1788) Durable sync's legacy `_meta` lane paged the responder's rows with a raw `ORDER BY ?g ?s ?p ?o` / OFFSET / LIMIT window that can end mid-subject. A graph-scoped author seal is one (g, s) subject with 14 `_meta` quads; when it straddles a page boundary and a round ends there (the deadline cutoff is between pages), the round delivers only a prefix. Because durable meta is verified per round with no cross-round accumulation, the batch-local control field `dkg:assertionVersion` is then never admitted (it arrives without its 13 seal siblings, or they arrive without it), so curated CGs permanently keep 13/14 quads and the curator can no longer resolve the author to VM-publish. Make the durable-meta responder emit pages that always END on a (g, s) subject boundary: a subject straddling the requested row limit is emitted in full (the page EXTENDS past `limit`, never trims). Each round ends on a page boundary and every page ends on a subject boundary, so each round's accumulated meta ends on a subject boundary, the checkpoint offset is subject-aligned, and a seal's rows are never split across a round. This mirrors the changelog lane's per-graph atomicity without coupling seal admission to a receiver-side store re-read. EXTEND (not TRIM) is required: trimming could empty a page when one subject fills the whole window, and an empty page is the requester's EOF signal. Extending is safe because durable meta uses byte-budget pagination (requester page size 8192 > the 500 legacy cap), so the requester never treats an over-sized page as EOF — it advances by the actual row count and the next OFFSET lands on the next subject. - graph-plan.ts: `metaSubjectKey` helper; `subjectAtomic` extend on the cached (snapshot) path; `readDurableMetaRowsPageSubjectAtomic` (+ complete-trailing- subject re-read) for the store-paged no-session / oversized-fallback lane. - Regression test asserts a seal straddling a page boundary materializes all 14 quads incl. `dkg:assertionVersion` (0-or-14 per page, no dup/gap) on both loaders; fails-before / passes-after confirmed. - Update the snapshot-cache anti-copy guard to allow the bounded one-subject boundary peek (no read before offset, no whole-snapshot iteration retained); rework two concurrent-interleaving cases whose single-subject data encoded the now-fixed cross-page split to use distinct subjects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…1788) The store-paged durable-meta subject-atomic wrapper completed a straddling trailing subject with a bound-subject re-read, which returns nothing for a blank-node subject — so it fell back to the raw limited window and silently split the subject across the page (and thus across sync rounds), the exact #1788 loss for a batch-local control predicate. A blank node cannot be paged atomically on this multi-query lane: Oxigraph relabels it per query (observed `_:x` -> `_:<hash>`), so its label AND sort position differ across the store-paged lane's separate queries and across sync rounds — no bound re-read or OFFSET continuation can re-identify it. Conforming writers only emit IRI `_meta` subjects (metadata generators build deterministic IRIs; the publisher rejects blank nodes), so a non-IRI subject here is non-conforming input reachable only via the unverified system-CG peer-ingest path. Refuse it (throw) rather than emit a subject-split page: the throw only ever fires on non-conforming meta, never in normal sync, and never silently corrupts a seal. - graph-plan.ts: throw on a non-IRI straddling subject in the store-paged lane (replacing the silent raw-window fallback); keep the exact bound-IRI re-read for IRI subjects; fail loud too if an admitted IRI re-reads empty. Updated the JSDoc with the peer-ingest reachability + blank-node label-instability rationale. - Added a round-based regression (pageThrough over a real store) proving a straddling blank-node subject THROWS rather than splits; the 3 IRI cases stay green. Fails-before/passes-after confirmed via a silent-split mutation. Root fix tracked in #1921 (enforce IRI-only durable-meta subjects at ingest), which makes this throw unreachable defensive code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ia growing-window extend (#1788) Supersedes the fail-loud throw: instead of refusing a straddling non-IRI `_meta` subject, the store-paged lane now serves it WHOLE, so no admitted subject (IRI or blank node) is ever split across a page — and thus across a sync round (the #1788 loss for a batch-local control predicate). `readDurableMetaRowsPageSubjectAtomic` completes the trailing subject by re-reading a GROWING window from the same offset (`limit + extra`, extra doubling) until the subject is fully contained (a later subject appears, or the store is exhausted), then cutting at the subject boundary. Each attempt is a SINGLE ordered query — the key property for blank nodes: Oxigraph relabels a blank node per query, so a subject-bound re-read or a multi-query paged loop could not re-identify one, but within one self-contained window its label (and therefore the (g,s) boundary comparison) is consistent, so the subject is served whole. This reuses readDurableMetaRowsPage (deletes the bespoke readDurableMetaSubjectRows query) and works uniformly for all admitted terms. The loop is bounded (the finite store exhausts the growing window); a pathologically large subject is emitted whole rather than truncated, with the transport frame limit as the final guard. - Replaced the blank-node throw test with a round-based atomic regression (pageThrough over a real store): a straddling blank-node subject materializes 0-or-all per page (never split), no throw, cursor continuity; the 3 IRI cases stay green. Fails-before/passes-after confirmed via a split mutation (both store-paged straddle tests fail without the extend). Defense-in-depth follow-up #1921 (enforce IRI-only durable-meta subjects at ingest) remains valid but is now non-urgent, since the extend already serves blank-node subjects atomically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…paging (#1788) The store-paged straddle tests only exercised the EOF completion branch (window shorter than requested). Add a case where the grown window is FULL and a later multi-row subject appears in it, so the cut must drop the later subject's partial rows exactly at the seal boundary — otherwise the page would emit a partial next subject (a #1788-class split of THAT subject). Asserts page 1 = 5 filler + 14 seal with zero later-subject rows, the later subject resumes intact on page 2, and both subjects are 0-or-all per page. A skip-the-cut mutation fails only this new case (the EOF case still passes), confirming it closes the coverage gap. Addresses otReviewAgent review on PR #1916. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t byte budget (#1916) The subject-atomic extend served a straddling subject WHOLE while durable meta was still serialized without a byte cap, so an oversized admitted `_meta` subject (reachable via the unverified system-CG peer-ingest path: one dkg:memoryLayer row + thousands of junk rows) produced a response exceeding the transport frame that repeatedly failed instead of making progress. Option D (bounded-extend + safe-split), preserving #1788: - Byte-cap the durable-meta serialization with serializeResponderRowsWithinByteBudget (mirror durable data) — the frame-safety backstop under everything else. - Bound the store-paged growing-window extend by the response byte budget: a single subject whose rows reach the budget is, by that size alone, provably not a valid seal/descriptor (those are ~KB — orders of magnitude below the 4 MiB budget: 10 MiB router cap minus 6 MiB headroom). The extend stops growing (bounding memory) and the byte-budget serialization splits that oversized subject across pages with forward progress. Splitting it is safe: it carries no batch-local control envelope the receiver admits atomically. - Every sub-budget subject (all real seals/descriptors) still ends on a (g,s) boundary and is served whole — the byte cap triggers only when ONE subject alone exceeds the budget. - Durable-meta cache: releaseOnShortPage=false so a byte-capped short page can't prematurely release the session and strand the rest of the meta (meta is byte-budget-paginated; EOF is an empty page, not a short one). Tests: (a) an oversized admitted subject stays under the budget with forward progress and a bounded fetched window (mutation-verified: fails without the growth-stop); (b) a realistically-large-but-VALID seal (max normal literals) is still served whole, never split; plus the existing blank-node-atomic, later-subject-cut, clean-boundary and no-dup/no-gap cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t≠EOF; unify response-byte model (#1916) Option A resolution of the round-5 review on the durable-meta byte-cap (no behavior change to the cap itself — it is safe for the whole testnet-canary+ deployment, where durable-meta pagination already treats a short page as non-EOF, only an empty page as EOF): - Document the pagination contract explicitly at both ends: the byte-cap (sync-handler) and the requester default (page-fetch) now state that durable meta uses byte-budget pagination where a SHORT page is NOT EOF, that this is a REQUESTER-SIDE default (syncPageSize > SYNC_PAGE_SIZE), not wire-negotiated, and that a byte-capped meta page is ~14000 rows (≫ the 500-row limit) so it is never short for normal meta. - Handler-level regression through registerSyncHandler: an oversized admitted _meta subject (>4 MiB via many sub-limit large literals) is served as byte-capped prefixes that each stay under the frame and page to completion — proving the durable-meta handler branch actually uses the budgeted serializer. Mutation-verified: reverting the handler to serializeResponderRows fails it (one 4.8 MiB over-budget response). - Unify the response-byte model (addresses the "two byte models" review): a shared serializedResponderRowByteLength / serializedRowsResponseBytes (N-Quads UTF-8 wire bytes) is now used by BOTH the byte-budget serializer and the subject-atomic extend's growth-stop; heap estimates remain for snapshot memory budgets only. Follow-up #1923 tracks wire-negotiating meta pageMode for the pre-testnet-canary short=EOF-requester + oversized-subject residual. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tiated meta byte budget (#1916) Round-6 fix. Two parts. PART 1 (correctness — a real #1788 reintroduction): the option-D byte cap was a ROW-prefix serializer, so "≤ budget" and "subject-atomic" were different guarantees and only the former held. Large-literal rows preceding a valid seal consumed the budget and the row-wise cut landed mid-seal → the seal split across rounds. Fix: readDurableMetaPage now returns a SUBJECT-ATOMIC, BYTE-FITTING page via `subjectAtomicBudgetEnd` — accumulate WHOLE (g,s) subjects up to the response byte budget (and the requested row limit), cut on a subject boundary, DEFER a subject that would not fit to the next page, and split ONLY a FIRST subject that alone exceeds the budget (byte-fitting row prefix — provably not a valid seal at that size). Applied UNCONDITIONALLY to both the cached (in-memory) and store-paged (growing-window) lanes, so every returned page is subject-atomic AND ≤ budget. Blank-node subjects stay grouped: subjects are keyed within ONE query's result (cached snapshot / single store-paged window), the same single-query self-consistency the #1788 fix relies on. PART 2 (wire contract — mirror durable data): request-build advertises `pageMode`/`pageRowsHint` for the durable-META phase (additive both directions; signed limit stays the 500-row legacy cap), and the responder gates the durable-meta byte-budget serializer on it. With PART 1 bounding every page ≤ budget, both the negotiated (byte-budget serializer) and legacy (plain serializer) branches are frame-safe and subject-atomic; the gate records the explicit contract instead of relying on the requester-side syncPageSize>500 default. Tests: large-literal fillers BEFORE a valid 14-row seal → the seal is deferred whole, never split (mutation-verified: a row-prefix cut fails it); both-branch handler test (negotiated + legacy) through registerSyncHandler → oversized subject stays ≤ frame and pages to completion in both; existing subject-atomic / blank-node / later-subject-cut / clean-boundary / oversized cases stay green. Follow-up #1923 (wire negotiation now in-PR; residual pre-canary short=EOF case + SyncPagePolicy refactor tracked there); root fix #1921. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… builder (#1916) The #1916 mirror extended the request builder's byte-budget page-mode advertisement from durable data to durable meta, but no test exercised the `phase === 'meta'` term through the real `buildSyncRequestEnvelope` — the handler-level tests hand-craft the `pageMode` field, so a regression dropping 'meta' from the `useByteBudgetPage` condition would silently break wire negotiation without failing any test. Add two focused request-builder cases: meta advertises `pageMode` + `pageRowsHint` above the 500-row legacy cap (signed limit stays capped for digest compatibility), and does not advertise at/below it. Mutation-verified: dropping 'meta' from the condition fails the above-cap case. Addresses review 3632353146. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…lent short page (#1788) Gate the durable-meta oversized-subject handling on the wire negotiation so a non-negotiated (pre-testnet-canary) requester can never receive a byte-fit SHORT page it would read as EOF — which silently drops the rest of the metadata and splits a seal (#1788 reintroduced). Reverses the earlier unconditional byte-fit; for a data-integrity fix, a loud failure beats a silent partial-metadata loss, and this touches only the legacy path (every testnet-canary+ requester negotiates byte-budget meta paging). Subject-atomic accumulation stays UNCONDITIONAL on both branches (#1788 holds for legacy too — never split a seal); only the oversized handling differs: - NEGOTIATED ('byte-fit'): unchanged, verified behavior — byte-cap the page, splitting a lone oversized FIRST subject to a byte-fitting prefix. The requester paginates via empty=EOF, so a short page is not EOF. - NON-NEGOTIATED ('fail-loud'): never byte-fit. Emit whole subjects up to the row limit; if the page still cannot be produced frame-safe — a single oversized subject OR the cumulative row-limit-bound page — throw DurableMetaPageFrameError, which the handler surfaces as a hard, non-retryable transport error (never an empty EOF body). Root-fixed at ingest by #1921. Implemented as an `oversizedPolicy` parameter on the shared subjectAtomicBudgetEnd accumulator (in 'fail-loud' mode with an unbounded row limit it returns the whole page unless it throws), threaded through readDurableMetaPage and the store-paged loader; sync-handler selects it from usesMetaByteBudget. Tests (mutation-verified — disabling the fail-loud branch fails the legacy oversized case): - cached-path large-filler-before-seal byte-fit (the cached-trim gap the store-paged test did not cover): the seal is deferred whole, never split; - legacy normal meta paginates in row-limit subject-atomic pages, short only at EOF, no throw; - legacy oversized _meta (single subject AND cumulative page) throws DurableMetaPageFrameError, never a silent short page; - handler-level: negotiated byte-caps + pages to completion; legacy fails loud. Addresses the escalated round-7 blocking review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

otReviewAgent

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

The otReviewAgent review of 80819ea failed with an operational error ("Business logic reviewer failed: retry_exhausted") rather than completing, so the gating fix has no genuine bot verdict yet. This empty, tree-identical commit re-fires the review event; the PR tree is unchanged from 80819ea (same gating code already verified in review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Clarify, at the class, that the fail-loud durable-meta failure is HARD and NON-RETRYABLE: the responder records it as outcome 'error' and re-throws it (never a retryable wrapper, never an empty EOF body) because a retry cannot make an oversized subject servable to a legacy requester — the only resolutions are a requester upgrade or the #1921 ingest fix. Also records why SYNC_BYTE_BUDGET_RESPONSE_BYTES (router read cap minus frame headroom) is the correct frame-safety threshold: it is the largest body guaranteed to fit one transport frame, so a page within it is always sendable. Comment-only; the gating logic is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Comment-only. Expand the loop-top guard comment in selectAdmittedMetadataIndexes to spell out why the IRI-only check must run BEFORE (not inside) the merkle/marker admission branch: indexIntegrityMetadata adds any merkleRoot-bearing subject to merkleSubjects without validating the term, and a persisted blank-node subject has no stable identity — the responder must later serve it back, where #1916's store-paged subject-atomic lane only stays sound via Oxigraph's per-query relabel. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…gest (#1921) (#1936) * fix(sync): reject non-IRI durable-meta subjects at unverified peer-ingest (#1921) Durable-sync _meta ingest funnels every peer-supplied metadata quad through selectVerifiedDurableSyncQuads, but neither meta selector (selectAdmittedMetadataIndexes, selectSystemOverrideMetadataIndexes) checked that the metadata SUBJECT is an IRI. A peer could inject a blank-node _meta subject -- including one bearing an integrity predicate such as dkg:merkleRoot -- and have it persisted into a durable _meta graph, where it has no trustworthy, stable identity. Add an isIriMetaSubject guard at the top of BOTH selector loops (before the merkle/marker admission branch), dropping non-IRI (blank-node / literal) subjects and counting them. REJECT, never skolemize: a peer-injected blank node has no authenticatable identity. The guard operates on incoming peer quads only -- no store read/DROP/sweep -- so there is zero data-loss risk to already-persisted data. A single aggregated warn is emitted per selection at all three guarded call sites, and the drop count is surfaced as droppedNonIriSubjects on DurableIntegritySelection (mirroring droppedSyncControlTriples). Also repoint the DurableMetaPageFrameError oversized-subject references from #1921 to #1923 (SyncPagePolicy): #1921 is IRI-only; a valid-IRI subject bearing a giant literal is an orthogonal size vector. The #1916 responder subject-atomic docstring is updated to note ingest now enforces IRI-only for NEW peer ingest, while the store-paged blank-node accommodation stays as defense-in-depth for pre-fix persisted data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(sync): clarify why the #1921 IRI guard precedes merkle-admission Comment-only. Expand the loop-top guard comment in selectAdmittedMetadataIndexes to spell out why the IRI-only check must run BEFORE (not inside) the merkle/marker admission branch: indexIntegrityMetadata adds any merkleRoot-bearing subject to merkleSubjects without validating the term, and a persisted blank-node subject has no stable identity — the responder must later serve it back, where #1916's store-paged subject-atomic lane only stays sound via Oxigraph's per-query relabel. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): advance meta cursor when a page is fully dropped as non-IRI (#1921) Addresses the otReviewAgent review of PR #1936. Bug (liveness): the durable requester advances the meta checkpoint for a metadata-only page that was ENTIRELY discarded only when the discarded rows are counted as deliberately consumed — durable-sync.ts gated that on droppedSyncControlTriples === totalFetchedMetaQuads. A page consisting solely of non-IRI `_meta` rows dropped by the #1921 ingest guard produced droppedSyncControlTriples=0, so updateMetaCheckpoint stayed false and the meta cursor pinned: the same page is re-fetched and re-dropped every round. This is also a regression vs pre-#1921 (those rows used to be persisted, so verifiedMeta>0 advanced the cursor). Reachable when a responder holds pre-fix persisted non-IRI rows or a malicious responder injects them on the wire. Fix: thread droppedNonIriSubjectTriples through the worker boundary (DurableBatchProcessResult -> wire result -> requester) and generalize the requester check to deliberatelyDroppedMeta = droppedSyncControlTriples + droppedNonIriSubjectTriples, compared to totalFetchedMetaQuads. This also fixes a MIXED all-discarded page (some controls + some non-IRI) that pins today because neither count alone equals the fetched total. Also folds in the review's other points: - Rename droppedNonIriSubjects -> droppedNonIriSubjectTriples (it counts dropped rows, mirroring droppedSyncControlTriples). - Add a literal-subject ingest-drop test (guard rejects blank-node AND literal). - Add requester-path regression tests (sync-requester-progress.test.ts) proving the meta cursor advances on an all-non-IRI page and a mixed all-discarded page; verified to fail without the propagate fix. Deferred: unifying the ingest/responder IRI term classifier -> #1940. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): stop non-IRI _meta subjects from authenticating durable data (#1921) Addresses the otReviewAgent review of PR #1936 (two 🔴). 🔴 Data-integrity: the #1921 admission-only guard dropped non-IRI metadata from PERSISTENCE but not from VERIFICATION. indexIntegrityMetadata added any subject bearing dkg:merkleRoot to merkleSubjects with no IRI check, so a peer's blank-node/literal subject became a verification candidate, self-consistently authenticated its bound DATA (the claimed root is peer-supplied, not on-chain-anchored), and got that data admitted — after which the selector guard dropped only the metadata, leaving orphaned, peer-forged data in the store (an injection vector). Empirically: a blank-node legacy envelope over matching data yielded dataIndexes=[0], metaIndexes=[]. Fix (candidate-gate): skip non-IRI subjects at the top of indexIntegrityMetadata's merkle/marker loop so they never become verification candidates and cannot authenticate data. A page bound only by a non-IRI envelope then has no verified descriptor and fails closed (rejected on a normal CG). The admission-selector guards + droppedNonIriSubjectTriples counting are kept unchanged (still load-bearing for persist-drop and meta-cursor advance). isIriMetaSubject is now null-safe: the requester's bounded-snapshot planner (planBoundedGraphScopedDurableBatch) runs indexIntegrityMetadata on raw fetched meta, which must tolerate malformed input rather than throw. Tests: - Flip the former "persisted blank-node" case to FAIL-CLOSED for BOTH a blank-node and a literal merkle-subject (rejected=1, dataIndexes=[], metaIndexes=[]); mutation-proven (removing the gate reverts to data-persists/meta-dropped and the tests fail). - Livelock-fix-intact: an acceptUnverified all-non-IRI metadata-only page is fully consumed (droppedNonIriSubjectTriples === total, not rejected) so the cursor still advances. - Worker-wire propagation (🔴#2): real processDurableBatchForWire on an all-non-IRI page reports the dropped count across selection->wire. Deferred: consolidating the requester's per-reason consumed-metadata counters into one worker-owned aggregate touches shipped sync-control checkpoint semantics -> tracked as a follow-up, kept out of this security PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(sync): consolidate durable meta-checkpoint accounting into one worker-owned count (#1921) Addresses the otReviewAgent review of PR #1936 (🟡: requester progress depended on per-reason verifier discard counters). The requester's discardedOnlyMetadataResponse summed reason-specific counters (droppedSyncControlTriples + droppedNonIriSubjectTriples) to decide whether a fully-discarded metadata-only page still advances the meta cursor. That coupled checkpoint orchestration to verifier discard policy, so each new deliberate-drop category would need another optional field threaded through the worker result plus another addend here. Consolidate: the WORKER (processDurableBatch) now emits a single reason-agnostic consumedUnpersistedMetaTriples = droppedSyncControlTriples + droppedNonIriSubjectTriples on DurableBatchProcessResult, and the requester's checkpoint predicate compares that one field to totalFetchedMetaQuads. The per-reason counters are kept as verifier-side diagnostics only and removed from the requester's structural contract. The already-shipped sync-control semantics are preserved bit-for-bit: for a pure-sync-control page nonIri=0 so the aggregate equals droppedSyncControlTriples exactly. Tests: - Worker-level: consumedUnpersistedMetaTriples === droppedSyncControlTriples + droppedNonIriSubjectTriples for pure-sync-control, all-non-IRI, and mixed pages. - Requester cursor-advance for pure-sync-control (regression guard for the shipped path), all-non-IRI, and mixed pages, all via the single aggregate; pin case when the aggregate < total. - Mutation-proven: neutralizing the worker sum breaks the all-non-IRI + mixed aggregate tests; neutralizing the requester's aggregate read breaks all three cursor-advance tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): sanitize non-IRI _meta out of durable verification, not just admission (#1921) Addresses the otReviewAgent 14:03 review of PR #1936 (a 🔴 nested on the resolved durable-integrity thread) + follow-up 🟡s. 🔴 Non-IRI metadata could still POISON verification. The candidate-gate stopped non-IRI subjects from becoming merkle/marker candidates, but metaBySubject was still built from ALL rows, and readIntegrityMetadata scans it for PART_OF (line ~604) — so a peer's `_:bad dkg:partOf "<valid-ual>"` row falsely invalidated the valid graph-scoped UAL → fatalUnscopedFailure → a VALID batch was rejected → durable sync pinned (a poison/DoS vector, opposite direction to the earlier authenticate-then-drop hole). Reproduced: valid descriptor + one non-IRI partOf row → rejected=1, dataIndexes=[], metaIndexes=[]. Fix (Option B, boundary sanitize): filter metaQuads to IRI-only ONCE at each verification entry and feed that to indexIntegrityMetadata + verifyLegacyCandidates, so non-IRI subjects can neither authenticate data (never a candidate) nor poison verification (never in metaBySubject or the raw legacy scan). This is applied in BOTH selectVerifiedDurableSyncQuads AND planBoundedGraphScopedDurableBatch — both call readIntegrityMetadata and were vulnerable. Admission (selectVerifiedQuads / the no-descriptor selectAdmittedMetadataIndexes) deliberately stays on the ORIGINAL metaQuads: the selectors still drop + count non-IRI rows (persist-drop and meta-cursor advance) and index into the original array. The verification outcome is subject-keyed, so no positional index remap is needed. The now-subsumed candidate-gate inside indexIntegrityMetadata is removed (single boundary invariant; addresses the "centralize the drop" 🟡). Also (net-new 🟡): make consumedUnpersistedMetaTriples REQUIRED (non-optional) on the requester's processed contract and drop the `?? 0` — it is the single checkpoint-advance signal, so every producer must set it (tsc-enforced) rather than silently reading 0 and pinning the cursor. Tests: - Valid graph-scoped descriptor + `_:bad dkg:partOf <ual>` → asset STILL verifies (rejected=0, dataIndexes=[0], full descriptor persisted), bad row dropped + counted; mutation-proven (neutralize the sanitize → rejected=1). - Direct selectSystemOverrideMetadataIndexes non-IRI test (rejected IRI candidate forces the accept-unverified override path + non-IRI rows → dropped + counted). - Updated the since-threading mock for the now-required aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(sync): make indexIntegrityMetadata IRI-precondition explicit; fix stale gate comment (#1921) Comment-only. Two follow-ups to the Option B boundary-sanitize: - indexIntegrityMetadata now documents an explicit PRECONDITION — callers MUST pass IRI-sanitized meta (selectVerifiedDurableSyncQuads + planBounded do; a new caller must too) — since the internal candidate-gate was removed. - Rewrite the selectAdmittedMetadataIndexes non-IRI guard comment: it claimed indexIntegrityMetadata "adds ANY merkleRoot-bearing subject" (the bot's cited drift), which is no longer true under the boundary sanitize. The guard's real job is admission drop+count on the ORIGINAL metaQuads (persist-drop + cursor accounting); non-IRI subjects can't reach the merkle/marker branch anymore. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(sync): cover the bounded-planner non-IRI partOf poison path (#1921) Addresses the otReviewAgent :181 🟡 — Option B's sanitize was added to the second verification boundary (planBoundedGraphScopedDurableBatch) but only the main selector path had a poison regression. Add a bounded-planner test: a valid 3-graph manifest + `_:bad dkg:partOf "<valid-ual>"` → the planner still projects the same safe prefix (safeNextOffset=8, 2 complete graphs) as the clean case, rather than returning null. Mutation-proven: removing planBounded's iriMetaQuads filter makes the plan null (poison invalidates the valid UAL) and this test fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(sync): let the verifier own the consumed-metadata checkpoint aggregate (#1921) Addresses the otReviewAgent :392 🟡 — consumedUnpersistedMetaTriples (the single signal the requester uses for meta-checkpoint advance) was computed one layer up in the worker by re-summing the verifier's diagnostic counters, so the worker had to know which discard reasons count toward requester progress. Move ownership to the verifier that classifies the drops: add consumedUnpersistedMetaTriples (REQUIRED) to DurableIntegritySelection, set at every return site as droppedSyncControlTriples + droppedNonIriSubjectTriples; the worker now TRANSPORTS verifiedSelection.consumedUnpersistedMetaTriples on the main path (no recompute), and its early-return branches (empty page / data-without-meta) still set 0 explicitly (tsc-enforced at every branch). The per-reason counts stay as diagnostics. Value-identical: same sum, computed where the drops are classified. Scope: only the count's OWNERSHIP moves. The larger prepareDurableMeta intake centralization (filter/drop/count of the verification-flow) stays deferred in #1943. Tests: added a durable-integrity assertion that selection.consumedUnpersistedMetaTriples === droppedSyncControlTriples + droppedNonIriSubjectTriples across pure-control / all-non-IRI / mixed pages; mutation-proven (drop the +nonIri addend → the all-non-IRI + mixed cases fail). Existing worker-aggregate + requester cursor-advance tests stay green (behavior unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>