fix(sync): reject non-IRI durable-meta subjects at unverified peer-ingest (#1921)

[Jurij89]

Summary

• Closes two peer-ingest attack vectors on durable _meta verification by sanitizing the verification inputs to IRI-only at BOTH entry points — selectVerifiedDurableSyncQuads and planBoundedGraphScopedDurableBatch now filter metaQuads through isIriMetaSubject into iriMetaQuads before indexIntegrityMetadata / verifyLegacyCandidates. (a) Candidate/self-auth: a non-IRI subject bearing dkg:merkleRoot can no longer enter merkleSubjects and self-authenticate its own peer-supplied data (the claimed root is peer-supplied, not on-chain-anchored). (b) Poison/DoS: a _:bad dkg:partOf "<valid-ual>" row can no longer reach readIntegrityMetadata's PART_OF scan and falsely invalidate a legitimate graph-scoped UAL (fatalUnscopedFailure → valid batch rejected → sync pins).
• Admission still drops + counts. Both meta selectors (selectAdmittedMetadataIndexes, selectSystemOverrideMetadataIndexes) deliberately run on the ORIGINAL metaQuads and keep an isIriMetaSubject guard at the top of the loop that drops and counts non-IRI rows into droppedNonIriSubjectTriples, emitting one aggregated warn per selection. Nothing non-IRI is ever persisted, and the drop is still accounted for the meta cursor. Verification is subject-keyed, so no positional remap is needed across the sanitized/original split.
• Livelock cursor-advance via one aggregate. An all-non-IRI meta-only page verifies zero meta; a naive drop that discarded these rows without counting them would leave updateMetaCheckpoint false and re-serve the same page forever (a livelock). The worker now exposes a single reason-agnostic consumedUnpersistedMetaTriples (= droppedSyncControlTriples + droppedNonIriSubjectTriples); when it equals totalFetchedMetaQuads the requester marks discardedOnlyMetadataResponse and advances the meta cursor. Per-reason counts remain verifier-side diagnostics only, keeping checkpoint orchestration decoupled from discard policy.
• Aggregate is REQUIRED, not optional. consumedUnpersistedMetaTriples is a required field on the requester's process-result contract — an optional field silently reading 0 would let the meta cursor pin with no type error, so every producer must set it.
• REJECT, never skolemize. A peer-injected blank node has no authenticatable, stable identity; skolemizing would mint a fabricated IRI over attacker-shaped metadata. Dropping is the only fail-closed option that preserves the invariant "every durable _meta subject is a conforming IRI".
• Incoming peer quads only — no store sweep. The guard operates on incoming quads; it never reads, DROPs, or sweeps the store, so there is zero data-loss risk to already-persisted durable data. A blank-node _meta subject written by a pre-fix peer may still reside in the store, so fix(sync): keep a graph-scoped seal's _meta subject-atomic across durable pages (#1788) #1916's per-query-relabel subject-atomic accommodation stays as defense-in-depth (docstring updated).
• Scope boundaries. The oversized-_meta-subject SIZE vector (a valid IRI subject bearing a giant literal) is orthogonal to this IRI-only guard and is deferred to Wire-negotiate durable-meta byte-budget pagination (pre-canary short=EOF requester + oversized meta subject) #1923 (SyncPagePolicy) — DurableMetaPageFrameError references repointed from Enforce IRI-only durable-meta subjects at unverified peer-ingest (root fix behind #1788 fail-loud) #1921 to Wire-negotiate durable-meta byte-budget pagination (pre-canary short=EOF requester + oversized meta subject) #1923. Unifying the ingest classifier isIriMetaSubject with the responder read's isIriTerm into one shared classifier is deferred to Unify the durable-meta IRI term classifier across ingest (durable-integrity) and responder read (graph-plan) #1940.

Related

• Fixes Enforce IRI-only durable-meta subjects at unverified peer-ingest (root fix behind #1788 fail-loud) #1921
• Root fix behind fix(sync): keep a graph-scoped seal's _meta subject-atomic across durable pages (#1788) #1916's fail-loud DurableMetaPageFrameError (blank-node reachability of the store-paged subject-atomic lane)
• Oversized _meta-subject SIZE vector (valid IRI subject bearing a giant literal) deferred to Wire-negotiate durable-meta byte-budget pagination (pre-canary short=EOF requester + oversized meta subject) #1923 (SyncPagePolicy)
• Deferred code-health: Unify the durable-meta IRI term classifier across ingest (durable-integrity) and responder read (graph-plan) #1940 (unify the ingest isIriMetaSubject with the responder read isIriTerm, and centralize the durable-meta intake filter/drop/count)

Diagrams

Flow 1 — Peer _meta-page verification (durable-integrity)

Before: a peer's non-IRI (_:blank / literal) _meta subject enters indexIntegrityMetadata over the full metaQuads, so it can either become a merkle candidate and self-authenticate its own peer-supplied data, or poison a legitimate graph-scoped UAL via readIntegrityMetadata's PART_OF scan (a valid batch is rejected and sync pins).

sequenceDiagram
    participant Peer
    participant Ver as Verifier durable-integrity
    participant Sel as AdmissionSelectors
    participant Store
    Peer->>Ver: metaQuads incl non-IRI _bad subject
    Ver->>Ver: indexIntegrityMetadata over FULL metaQuads
    Note over Ver: _bad merkleRoot enters merkleSubjects as a candidate
    Ver->>Ver: readIntegrityMetadata PART_OF scan sees _bad partOf valid-ual
    alt _bad bears merkleRoot
        Note over Ver: peer-supplied root self-authenticates _bad data
        Ver->>Sel: selectAdmittedMetadataIndexes over metaQuads
        Sel-->>Ver: admitted indexes incl _bad-authenticated data
        Ver->>Store: persist attacker data and meta
    else _bad bears partOf valid-ual
        Note over Ver: fatalUnscopedFailure rejects the VALID batch, sync pins
    end

Loading

After: both verification entry points (selectVerifiedDurableSyncQuads and planBoundedGraphScopedDurableBatch) filter to iriMetaQuads = metaQuads.filter(isIriMetaSubject) before indexIntegrityMetadata / verifyLegacyCandidates, so a non-IRI row can neither authenticate nor poison; admission still runs on the ORIGINAL metaQuads to drop and count it.

sequenceDiagram
    participant Peer
    participant Ver as Verifier durable-integrity
    participant Sel as AdmissionSelectors
    participant Store
    Peer->>Ver: metaQuads incl non-IRI _bad subject
    Ver->>Ver: filter isIriMetaSubject to iriMetaQuads
    Note over Ver: _bad removed before any indexing
    Ver->>Ver: indexIntegrityMetadata over iriMetaQuads
    Ver->>Ver: readIntegrityMetadata PART_OF scan never sees _bad
    Note over Ver: _bad can neither authenticate data nor poison the valid UAL
    Ver->>Sel: selectAdmittedMetadataIndexes over ORIGINAL metaQuads
    Note over Sel: isIriMetaSubject guard drops and counts _bad as droppedNonIriSubjectTriples
    Sel-->>Ver: admitted indexes without _bad
    Ver->>Store: persist only verified IRI data and meta

Loading

Flow 2 — Meta checkpoint advance on a fully-discarded meta-only page (durable-sync)

Before (naive drop): dropping the non-IRI rows from Flow 1 without counting them makes an all-non-IRI meta-only page yield verifiedMeta=0 with no consumed-count, so updateMetaCheckpoint stays false, the meta cursor pins, and the responder re-serves the same page — a livelock. (This is the naive form of the drop, not pre-PR main — where the page had no guard and was simply persisted; the diagram shows why the drop must be counted.)

sequenceDiagram
    participant Peer
    participant Req as Requester DurableSync
    participant Ver as Verifier
    participant CP as Checkpoint
    Peer->>Req: meta-only page all non-IRI subjects
    Req->>Ver: processDurableBatch
    Ver-->>Req: verifiedMeta 0 and no consumed-count
    Note over Req: naive drop -- non-IRI rows discarded but NOT counted, discardedOnlyMetadataResponse false
    Note over Req: updateMetaCheckpoint false, meta cursor pins
    Req->>Peer: re-fetch same meta offset
    Peer-->>Req: responder re-serves the same page, livelock

Loading

After: the worker exposes a single consumedUnpersistedMetaTriples aggregate (= droppedSyncControlTriples + droppedNonIriSubjectTriples); when it equals totalFetchedMetaQuads the requester marks discardedOnlyMetadataResponse and advances the meta cursor.

sequenceDiagram
    participant Peer
    participant Req as Requester DurableSync
    participant Ver as Verifier
    participant CP as Checkpoint
    Peer->>Req: meta-only page all non-IRI subjects
    Req->>Ver: processDurableBatch
    Ver-->>Req: verifiedMeta 0 and consumedUnpersistedMetaTriples equals controls plus non-IRI
    Note over Req: consumedUnpersistedMetaTriples equals totalFetchedMetaQuads, discardedOnlyMetadataResponse true
    Note over Req: updateMetaCheckpoint true
    Req->>CP: setCheckpoint cg meta nextOffset
    Note over CP: meta cursor advances, page consumed once, no livelock

Loading

Files changed

File	What
packages/agent/src/sync/durable-integrity.ts	Filter verification inputs to iriMetaQuads at BOTH entry points (selectVerifiedDurableSyncQuads + planBoundedGraphScopedDurableBatch); verifyLegacyCandidates now takes iriMetaQuads. isIriMetaSubject guard + droppedNonIriSubjectTriples count at the top of both admission selectors (running on ORIGINAL metaQuads); logDroppedNonIriMetaSubjects warn wired at all three selection call sites; new droppedNonIriSubjectTriples field on DurableIntegritySelection; new isIriMetaSubject helper. indexIntegrityMetadata no longer filters internally — precondition documented.
packages/agent/src/sync/requester/durable-sync.ts	Depend on the required consumedUnpersistedMetaTriples aggregate for discardedOnlyMetadataResponse / meta-cursor advance (replaces the droppedSyncControlTriples-only condition); field made required (not optional) on the process-result contract.
packages/agent/src/sync-verify-worker.ts	Add droppedNonIriSubjectTriples (diagnostic) and consumedUnpersistedMetaTriples (single checkpoint-advance signal) to DurableBatchProcessResult.
packages/agent/src/sync-verify-worker-impl.ts	Populate droppedNonIriSubjectTriples and the consumedUnpersistedMetaTriples aggregate (= dropped sync controls + non-IRI) across every processDurableBatch / processDurableBatchForWire result branch.
packages/agent/src/sync/responder/graph-plan.ts	Repoint DurableMetaPageFrameError oversized-subject docstring + message from #1921 to #1923; update the #1916 subject-atomic docstring — NEW ingest now enforces IRI-only but does not sweep the store, so the per-query blank-node relabel stays as defense-in-depth for pre-fix persisted data.
packages/agent/test/sync-control-metadata-admission.test.ts	New tests: forged blank-node merkleRoot dropped without persisting; non-IRI subject with merkleRoot cannot authenticate its data (blank + literal, fail closed); descriptor-less :401 branch drop; all-non-IRI page counted as fully consumed so the cursor advances; conforming IRI untouched; literal subject dropped; graph-scoped asset still verifies with a non-IRI dkg:partOf poison row present; system-override path drop.
packages/agent/test/sync-durable-worker-wire.test.ts	New wire tests over the real selection → processDurableBatchForWire path: pure-control, all-non-IRI, and mixed pages report the correct consumedUnpersistedMetaTriples (= sum of per-reason counts).
packages/agent/test/sync-requester-progress.test.ts	Switch stubs from droppedSyncControlTriples to consumedUnpersistedMetaTriples; add cursor-advance tests for an all-non-IRI page and a mixed fully-discarded page.
packages/agent/test/durable-sync-since-threading.test.ts	Set the now-required consumedUnpersistedMetaTriples in the process-result stub.
packages/agent/test/rootless-durable-bounded-progress.test.ts	New regression for the second (bounded-planner) poison path: a valid graph-scoped manifest + a non-IRI dkg:partOf poison row → planBoundedGraphScopedDurableBatch still returns the safe prefix (not null).

Test plan

• npx vitest run sync-control-metadata-admission — fail-closed auth for BOTH a blank-node and a literal dkg:merkleRoot subject (rejected=1, nothing persisted); a graph-scoped asset STILL verifies with a non-IRI dkg:partOf poison row present (rejected=0, full descriptor persisted, bad row dropped+counted); system-override-path non-IRI drop; conforming IRI untouched
• npx vitest run --config vitest.unit.config.ts durable/sync blast radius green (durable-integrity, sync-responder, sync-verify, sync-durable-worker-wire, sync-requester-progress, rootless-durable-bounded-progress)
• Bounded-planner poison regression (rootless-durable-bounded-progress.test.ts): a valid graph-scoped manifest + a non-IRI dkg:partOf row → planBoundedGraphScopedDurableBatch returns the safe prefix, not null
• Mutation-proven: neutralize the verification-input sanitize → the poison batch is rejected (rejected=1, the reproduced pre-fix bug); neutralize the consumedUnpersistedMetaTriples aggregate → the all-non-IRI + mixed cursor-advance tests fail (the meta cursor pins)
• tsc --noEmit clean (clean rebuild)
• No localhost_contracts.json / deployments/localhost/* / agent-docs/ in the diff

[otReviewAgent]

🟡 Issue: Centralize the durable metadata term classifier instead of mirroring it by comment

What's wrong
This PR makes the IRI-only durable _meta invariant span ingest and read paths, but implements the same string-prefix classifier in two private places. That is a boundary-cleanliness problem: future changes to RDF term encoding, skolemization, or literal handling now have to update two large files in lockstep, and reviewers have to remember that these private helpers are semantically coupled.

Example
isIriMetaSubject('') returns false, while the responder's private isIriTerm('') returns true. Even if an empty term is not expected, this already shows the two copies can drift because there is no single term-boundary helper.

Suggested direction
Extract a canonical isIriTerm/isDurableMetaIriSubject helper in the layer that owns the string-encoded Quad contract, then import it from both ingest and responder code. The important invariant should be shared by code, not by a comment pointing at a private function.

For Agents
Move RDF string-term classification into a shared sync/storage utility, use it from durable-integrity and graph-plan, preserve the current blank-node/literal handling, and cover the shared helper with small tests for IRI, blank-node, literal, and empty-string cases.

[Jurij89]

Deferring as a tracked follow-up: #1940.

The two helpers share the same _: / " prefix contract; the empty-string divergence you flagged (isIriMetaSubject('') → false vs the responder's isIriTerm('') → true) is a deliberate semantic rule, not drift: the ingest guard fails closed on a malformed empty _meta subject, while the responder read path never sees an empty term post-ingest. Unifying the two would change the responder read-path's empty-string handling and warrants its own review/tests, so I filed #1940 to track extracting a single canonical classifier rather than widen #1921's scope.

[otReviewAgent]

💡 Suggestion: Clarify the RDF term boundary instead of calling this a full IRI check

Why it matters
The helper is now part of the durable ingest contract, so an imprecise name and duplicated implementation make future boundary work easy to misread or accidentally diverge.

Suggestion
Either extract/reuse a shared isNamedNodeTermString-style helper with the responder, or rename/comment this helper to match the actual predicate. If durable _meta truly requires validated IRIs, use the canonical IRI validator at this boundary instead of a prefix-only test.

[Jurij89]

Deferring the shared-classifier extraction / naming to #1940 (durable-meta IRI term-classifier unification across ingest + responder). The prefix-only isIriMetaSubject predicate is intentional here: it matches the responder's isIriTerm contract (_:/" prefixes), and the empty-string divergence is a deliberate fail-closed hardening at ingest. Extracting a canonical shared helper / using a full IRI validator at this boundary is a boundary-cleanliness change that touches the responder read path too, so it deserves its own review + tests rather than widening this security PR. Tracked in #1940.

[lupuszr]

Final current-head review — c12e6c3

I completed another review of the current head and tested it as a virtual merge with current testnet-canary (3ca9212c).

No blocking findings. Recommendation: merge promptly.

The security/correctness path is now coherent:

• Non-IRI peer _meta rows are removed before both verification entry points (selectVerifiedDurableSyncQuads and planBoundedGraphScopedDurableBatch), so they can neither become a Merkle candidate that admits attacker-shaped data nor poison a valid UAL through a forged dkg:partOf row.
• Admission still runs over the original metadata array, preserving source indexes while dropping and counting every non-IRI row before persistence.
• consumedUnpersistedMetaTriples is required across the worker boundary. The requester advances a fully discarded metadata-only page only when the batch is otherwise clean and the aggregate exactly equals totalFetchedMetaQuads; partially explained or verification-rejected pages remain pinned. This closes the new drop-induced cursor livelock without creating an unsafe checkpoint advance.

Verification performed

• Clean virtual merge with current canary despite the PR branch being 7 base commits behind.
• Full agent/dependency build passed, including TypeScript type tests and package-root checks.
• Changed test files: 74/74 passed.
• Adjacent durable verifier tests: 49/49 passed.
• Responder admission parity: 12/12 passed.
• Total focused tests: 135 passed.
• git diff --check passed.
• GitHub currently reports MERGEABLE / CLEAN; Windows SQLite lifecycle and SPARQL scalability checks passed.

Nonblocking follow-ups

• isIriMetaSubject is the repository's string-term kind check (_: / " rejection), not full syntactic IRI validation. Shared classification/intake cleanup is appropriately tracked in Unify the durable-meta IRI term classifier across ingest (durable-integrity) and responder read (graph-plan) #1940 and Centralize durable-meta sanitize orchestration into a single prepareDurableMeta helper #1943; I would not widen this security PR for those refactors.
• The worker currently computes consumedUnpersistedMetaTriples from the two verifier diagnostic counters. The current calculation is correct and covered, but the aggregate should eventually be owned by durable-integrity (or a typed metadata-disposition result) so a future third discard reason cannot be omitted accidentally.
• This intentionally prevents new non-IRI ingestion and does not sweep previously persisted rows. Existing-store audit/remediation, if required, should be operationally scoped rather than added to this PR.

Given the verified attack-path closure, green merged-state tests, and absence of a current correctness blocker, I recommend merging #1936 promptly rather than waiting for the follow-up refactors.

[otReviewAgent]

🟡 Issue: Stop hand-maintaining the consumed metadata aggregate in every return branch

What's wrong
The PR introduces a useful aggregate boundary, but the implementation keeps the invariant as repeated arithmetic across several branches. That preserves incidental complexity inside the verifier and makes the next discard reason or branch edit more fragile than necessary.

Example
The invariant is currently repeated as selectedMetadata.droppedControls + selectedMetadata.droppedNonIriSubjectTriples at lines 449, 1075, and 1116, while early returns manually spell out all-zero counters. Adding another consumed-but-unpersisted reason means every return branch has to be audited again.

Suggested direction
Have the metadata-selection helper return a drop-count object with a derived consumedUnpersistedMetaTriples, or build DurableIntegritySelection through a small constructor. The caller should pass counts, not restate the aggregate formula branch by branch.

For Agents
In packages/agent/src/sync/durable-integrity.ts, introduce a DurableMetaDropCounts/MetadataAdmissionSelection helper or result constructor that computes consumedUnpersistedMetaTriples exactly once. Preserve the existing public fields and requester behavior; the existing aggregate tests should prove the helper is wired through every branch.

[Jurij89]

Deferring to #1943 — this is the same centralization theme it already tracks. #1943 covers a prepareDurableMeta / DurableMetaDropCounts helper (or a DurableIntegritySelection constructor) that computes consumedUnpersistedMetaTriples exactly ONCE and threads it through every return branch, so a new discard reason won't require auditing each branch.

The aggregate-OWNERSHIP move (the verifier owns the field, single requester signal) shipped in this PR (b7710dc). Computing-it-once across the 7 return sites is the deferred helper refactor: it reworks the durable-meta verification INTAKE on the security-critical path, so it belongs in #1943 with its own design + mutation-proof + review rather than widening this security fix. Tracked in #1943; leaving this thread open.