fix(sync): serve TTL-filtered SWM meta past the 64,000-row snapshot ceiling

[branarakic]

Symptom

A context graph whose SWM _meta snapshot reaches 64,000 rows becomes permanently unsyncable on the SWM meta lane — to every peer, forever. Live on mainnet: 10 of 15 cores emitting Sync responder snapshot memory budget … phase=meta, workspace=true, reason=snapshot_rows, rows=64001 78–202×/day; the canonical victim is fifa-world-cup-2026 (≈4,600 data quads, 64,001+ meta rows), which reports data=0 sharedMemory=0 denied=0 indefinitely while query-remote proves the data exists. The requester's page-size retry ladder is provably futile because the cap is on snapshot build, not page size.

Root cause

packages/agent/src/sync/responder/graph-plan.ts:504 — readSwmMetaPage passed params.cutoffIso == null positionally as fallbackOnPerSnapshotBudget into readResponderRowsPage (graph-plan.ts:1687, enforced at :1702). TTL-filtered sessions — the normal modern path — therefore got a bounded refusal with no fallback. Compounding it, readBoundedSwmMetaSnapshot applied the 64,000-row / 32 MiB budget to the raw meta graph before the TTL filter, so the refusal fired even when the fresh subset a TTL session actually needs was a few dozen rows. The cap is Math.min-clamped (snapshot-cache.ts:150-152), so no configuration can raise it.

Why the naive fix melts stores

The gate was deliberate. The TTL-filtered fallback query was SELECT DISTINCT ?g ?s ?p ?o + a six-predicate UNION join + global ORDER BY ?g ?s ?p ?o re-evaluated with a growing OFFSET per page over a mutable graph family — O(N²) paging of the #1597 listGraphs-storm class, able to pin multiple cores and gigabytes until the HTTP timeout. Flipping the flag alone would trade a deterministic bounded refusal for a store-melting query on exactly the mainnet cores currently refusing. This PR deletes that query instead of reviving it (and removes the dead readSwmMetaRows twin), and only then enables the fallback.

Bounded design

Mirrors how the SWM data lane solved the identical problem (buildFreshSwmDataGraphPlan):

1. buildFreshSwmMetaPlan — per meta graph, two small-result discovery queries (no payload rows, no sort, no OFFSET): fresh subjects by dkg:publishedAt >= cutoff (the accepted readFreshSwmRoots shape), and graph-scoped SWM heads via the fresh-WorkspaceOperation tuple join (heads deliberately carry no own publishedAt; they are admitted with their immutable commitment atomically). Chunked VALUES ?s … GROUP BY ?s counts complete a session-cached plan of graph/subject/count scalars only (createResponderFreshSwmMetaPlanMemo, same lifetime/refresh contract as the data-lane plan memo).
2. readBoundedFreshSwmMetaSnapshot — the snapshot now materializes only the admitted rows, so the per-snapshot budget binds on what is actually served. The fifa class (huge history, small fresh subset) takes the ordinary memoized-snapshot path. Final admission still runs through filterSwmMetaSnapshotRows, the canonical in-process filter.
3. readFreshSwmMetaRowsPageFromPlan — if even the admitted set exceeds the budget, the session degrades to whole-subject window pages walked over the plan's prefix sums: VALUES ?s-anchored reads with no store-side ORDER BY and no OFFSET (rows sorted in-process; plan subject order is compareCodePoint, identical to compareRows on ?s). Chunk row totals are verified against the plan, so a mutated subject fails the session (requester restarts with a fresh plan) instead of skipping/duplicating rows.
4. Legacy cutoff-less sessions keep the existing raw-snapshot + unfiltered store-paged compatibility path unchanged (pinned by test).
5. The one remaining bounded refusal: a single subject exceeding the hard 64,000-row build cap — a coherent row-group that cannot fit any budget, impossible via organic operation history, pinned by test.

#1788 interaction (split seal row-groups stripping dkg:assertionVersion)

The SWM requester (fetchSyncPages → processSharedMemoryBatch) reassembles all meta pages before verification, so page boundaries cannot strip fields in this lane — proven by a new requester-side test through real fetchSyncPages with 4-row pages against 5- and 11-row groups, on both the snapshot and plan-paged lanes. Additionally, the plan lane reads each seal/head subject atomically within one chunk query, so unlike the durable batching in #1788 a row-group can never be torn even responder-side. (The durable-lane batch split itself is #1788's scope, not inherited here.)

Test evidence

New packages/agent/test/sync-responder-swm-meta-ceiling.test.ts (8 tests, added to the explicit vitest.unit.config.ts include list), seeding real >64,000-row Oxigraph stores at default production budgets:

scenario	result
fifa shape: 64,026 raw rows, 26 fresh	served completely in 4 pages, 22 ms (was: permanent refusal)
intrinsically oversized fresh set: 65,000 admitted rows	served completely in 14 bounded pages, 774 ms, no refusal
plan-paged vs snapshot lane	set-equivalent across buckets, heads, stale exclusion
subject mutates between pages	session fails loudly, fresh session recovers fully
single 64,001-row subject	bounded refusal (the only one left)
legacy cutoff-less lane	unchanged store-paged fallback, no TTL join
requester reassembly (snapshot + paged lanes)	every op and seal/head row-group complete; dkg:assertionVersion never stripped

Mutation-tested (each mutation applied, run, reverted): reintroducing the positional params.cutoffIso == null fails exactly the 4 fallback-dependent tests (the fifa-shape test still passes — budget-on-filtered-set fixes that class independently); disabling count-verification fails exactly the mutation-detection test; removing the single-subject cap fails exactly the pathological-subject test; an off-by-one in the window slice fails all 4 paged-lane correctness tests.

Gates: packages/agent unit suite 1113/1113 green; tsc --noEmit clean; sparql-scale-lint --diff 0 new blocking (the rewritten legacy query carries reviewed R2/R3 pragmas; all new plan queries are lint-clean by shape). A 2-node devnet repro was not run: the devnet port map collides with a live mainnet edge node on this machine; the handler-level + fetchSyncPages tests above exercise the full responder/requester protocol path with real stores at real budgets, which is the same repro minus libp2p transport.

Mainnet impact

Unblocks the entire ≥64k-meta CG class (fifa-world-cup-2026, start-systems, and every long-lived active CG drifting toward the ceiling) alongside the already-merged materialization fix, and silences the 78–202/day refusal storms on 10/15 cores. TTL sessions get strictly cheaper responder work (indexed discovery + exact-subject reads instead of raw-graph materialization). Complementary _meta growth bounding (TTL/compaction of operation history) remains open as the cause-side follow-up.

Fixes #1847

🤖 Generated with Claude Code

[Jurij89]

Adversarial review — fix(sync): serve TTL-filtered SWM meta past the 64,000-row snapshot ceiling

Reviewed against the PR head (b7ba6d2bb). This is a strong, well-engineered fix and it correctly removes the #1847 permanent-refusal class. I checked the parts most likely to hide a paging bug and they hold up:

• The prefix-sum paging invariant is sound. compareRows = compareCodePoint(g,s,p,o) (graph-plan.ts:434) and the plan sorts subjects with compareCodePoint(s), so within a single-graph window the in-process row order is exactly the plan's subject order — the slice(skip - windowStart, …) cursor lands correctly, with no skips/dupes across pages or across graph entries.
• The deleted query genuinely was the store-melter (DISTINCT + 6-predicate UNION + global ORDER BY + growing OFFSET over a mutable graph family), and the legacy cutoffIso == null path is preserved unchanged. Deleting-rather-than-reviving is the right call.
• The realistic oversized cases degrade gracefully: the row-count budget throws a snapshot_rows SyncRowSnapshotBudgetError → readResponderRowsPage falls back to the bounded plan-paged reader; the cumulative byte budget throws snapshot_bytes the same way.

Only one finding survived adversarial verification, and it's LOW.

---

🟢 LOW — the new fresh-meta window reader is the one byte-limited responder query that doesn't convert StoreResponseTooLargeError into a graceful budget refusal

readFreshSwmMetaSubjectWindowRows (graph-plan.ts:2389-2428) issues its 100-subject chunk query with maxResponseBytes = snapshotResponseByteLimit(SYNC_RESPONDER_SNAPSHOT_BUILD_MAX_BYTES_ESTIMATE) (≈64 MiB after the ×2 wire headroom) but has no try/catch. Since readResponseTextBounded throws StoreResponseTooLargeError on overflow (http-response-limit.ts:44), a byte-overflowing chunk escapes uncaught through both consumers (readFreshSwmMetaRowsPageFromPlan:2487, readBoundedFreshSwmMetaSnapshot:2535). readResponderRowsPage only degrades on isPerSnapshotBudgetError (:1803), which is instanceof SyncRowSnapshotBudgetError — StoreResponseTooLargeError is a different class, so it's re-thrown, and sync-handler.ts:899-900 reports it as outcome 'error' with a raw (non-QuietRetryable) rethrow rather than the graceful 'limit' path the row-cap and every sibling produce.

This is a consistency gap: it's the only byte-limited responder query missing this conversion — the four siblings all do it (graph-plan.ts:774 durable manifest, :1542 exact-graph, :2100 the legacy SWM-meta twin readBoundedSwmMetaSnapshot, :2930 durable-meta twin).

Why LOW, not higher: the trigger is pathological. SWM meta literals are size-bounded (oversize-filter.ts — meta graphs "never legitimately hold large literals"), so reaching a 64 MiB single-chunk body needs ~1,100+ near-max literals concentrated in one 100-subject chunk, or a single ~64,000-row subject that still slips under the per-subject cap. Not something organic operation history produces. But the fix is trivial and removes an asymmetry that could bite later.

Fix: wrap the store.query in readFreshSwmMetaSubjectWindowRows in the same try/catch the legacy twin already uses at :2100 — on StoreResponseTooLargeError, throw snapshotBudgetError({ reason: 'snapshot_bytes', … }) so it degrades through the existing isPerSnapshotBudgetError path. (If a byte-overflowing chunk should actually be served rather than refused, the window reader would additionally need to re-chunk by byte total, not just the 100-subject count — but a coherent bounded refusal is already strictly better than today's raw transport error.)

Note: the new test suite runs on the embedded OxigraphStore, which ignores maxResponseBytes, so this branch is entirely untested — worth an HTTP-store (or a low-maxBytesEstimate) case alongside the fix.

---

Two things I checked that are not defects (noting for the record)

• The paged lane returns plan rows without filterSwmMetaSnapshotRows, while the snapshot lane applies it. I initially flagged this as lane-dependent output, but it verifies as benign: the plan discovery and the filter admission agree for canonical typed-literal meta writes (the plan is a superset only in principle), and the requester re-verifies the reassembled meta regardless. One suggestion: the set-equivalent across buckets test only seeds subjects where plan-admission == filter-admission, so it can't catch a future drift where the filter does exclude a plan-admitted subject. A test that seeds such a subject would lock the "both lanes serve the same set" invariant the design relies on.
• The single-subject > SYNC_RESPONDER_SNAPSHOT_BUILD_MAX_ROWS refusal is intentional, documented, and strictly better than the prior graph-level refusal — correctly the only bounded refusal left.

---

Method: 4 lens-scoped finders (paging correctness / lane divergence / budget+error paths / deleted-query + SPARQL + #1788 + tests) plus one adversarial refuter per candidate; 8 of 9 candidates were refuted (benign asymmetries, intentional behavior, or test-coverage-only observations). The surviving finding and the paging invariant were re-verified by hand.

[lupuszr]

Changes requested on b7ba6d2bb71f7e2fc5962f3fd425186fbe777336.

Two merge-blocking issues remain:

1. The plan-paged lane validates only subject row counts. A same-count delete+insert between pages passes the guard and can assemble a hybrid subject. A direct five-row/page-size-one reproduction completed without throwing, retained an old row and a replacement row, and omitted dkg:publishedAt. Mutable SWM heads normally update without changing row count, so requester reassembly alone does not make the subject atomic.
2. The fresh-subject plan is not covered by any row, byte, or global responder-memory budget. Discovery materializes every admitted subject, and up to 64 peer-scoped copies can remain cached for ten minutes. With the default 30-day SWM TTL, a large admitted set can exhaust heap before payload paging begins.

I also confirmed the existing store-response-limit finding: StoreResponseTooLargeError escapes the snapshot path untyped, so readResponderRowsPage does not enter its per-snapshot fallback. Existing thread: #1868 (comment)

Validation: runtime package build passed; 51 focused responder tests passed, including all eight PR tests; two targeted reproductions confirmed the same-count hybrid and untyped store-response failures.

[branarakic]

Review dispositions — all four 🔴 fixed in bb50dc541, structural follow-through in a92e14c30. Every fix is mutation-tested (mutant applied → exactly the intended test fails → reverted), sparql-scale-lint --diff reports 0 new blocking findings, and the full agent unit suite is green (89 files / 1123 tests).

🔴 graph-plan.ts:2273 — subject plan unbounded, bypasses responder budgets (@lupuszr) — FIXED, agreed this was the most serious one. Bounded by construction now, plus global accounting:

• Both discovery queries carry LIMIT (remaining subject allowance + 1) and the fixed snapshot-build response byte cap; crossing either is a typed per-snapshot refusal (FRESH_SWM_META_PLAN_MAX_SUBJECTS = 32_000, cumulative across the phase's graphs — every admitted subject serves ≥1 row, so sessions go plan-paged long before this cap can bind organically). The count queries got the same response cap. A bounded refusal instead of an unbounded control-plane plan, exactly as suggested.
• The plan now computes a retained-bytes estimate (also capped by the fixed 32 MiB build constant, bounding pathological IRI lengths), and the memo charges it to the process-wide SyncResponderSnapshotBudget as a control-plane entry: global rows/bytes apply (LRU-evicts idle plans, typed global_bytes rejection under pressure = the quiet retryable limit), while the operator/test-shrinkable per-snapshot caps deliberately do NOT — shrinking those is how a session is forced into plan-paged mode, and rejecting the plan there would turn the degradation back into a bug(sync): SWM meta lane permanently unsyncable once _meta reaches 64,000 rows (oversized-snapshot fallback disabled for TTL-filtered sessions) #1847-class refusal.
• Tests: cardinality-cap refusal (asserts the cap-derived LIMIT reaches the store), memo accounting unit tests (admit/LRU-evict/global-reject, per-snapshot exemption, eviction releases charge), and a handler-level wiring test that fails if registerSyncHandler stops passing the budget into the plan memo. Mutants killed: cap check removed, LIMIT removed, memo admission disabled, handler wiring removed — each by exactly the intended test.

🔴 graph-plan.ts:2420 — added === expected passes same-count mutations, tears subjects across pages (@lupuszr) — FIXED with two-layer binding in readFreshSwmMetaSubjectWindowRows:

• Per-subject row counts vs the plan (the old check was per-window aggregate — compensating cross-subject mutations inside one window passed it and misaligned every later prefix-sum slice, duplicating/skipping at page seams).
• A content digest (sha-256 over the compareRows-sorted p/o pairs, length-prefixed) bound on a subject's FIRST window read of the session and verified on every REREAD — your exact repro (one 5-row op, page size 1, same-count replacement between pages) now fails the session instead of assembling the hybrid, reproduced both at the responder and through the full fetchSyncPages requester path.
• One deliberate narrowing, made explicit in code and tests: a subject read exactly ONCE (never split across pages) that mutates same-count before its only read serves the newer coherent whole group — bounded freshness skew, like any keyset pager, never a torn hybrid. Binding content at plan build would require reading all rows up front, i.e. the unbounded materialization finding Game coordinator gossip hardening (PR #29 follow-up) #1 exists to kill. A store snapshot would give strict serialization but isn't available on all backends; digest-on-reread is the strongest bounded option. Test asserts the whole-group/no-tear property explicitly.

🔴 graph-plan.ts:2405 — store byte-limit errors bypass the TTL meta fallback — FIXED as suggested: readBoundedFreshSwmMetaSnapshot converts StoreResponseTooLargeError from window materialization into snapshotBudgetError({ reason: 'snapshot_bytes' }), so the phase degrades to the plan-paged reader. The conversion is deliberately NOT applied inside readFreshSwmMetaRowsPageFromPlan's own bounded window reads, so a genuinely oversized single page still surfaces hard rather than being masked. Regression: store-cap thrown for many-subject window queries only → session completes via plan paging (mutant reverting the conversion fails exactly this test). Plan discovery/count queries convert the same error into typed refusals at build time.

🔴 test:293 — mutation coverage only proves row-count changes — FIXED: same-count replacement of a split subject (responder-level + full requester path), compensating cross-subject count mutation, and the coherent-new-group skew case, alongside the existing grow-mutation test. Mutants for digest-off and aggregate-count-only are each killed by their dedicated test.

🟡 :607 — positional fallback boolean — DONE: readResponderRowsPage takes a named ResponderRowsPageOptions object ({ loadSnapshot, fallbackOnPerSnapshotBudget }); all call sites updated, policy spelled out at the TTL call site. Reintroducing the original params.cutoffIso == null defect through the named field kills 9 tests including the documented oversized-fresh kill.

🟡 :2542 — byte-budget fallback unverified — DONE: dedicated regression exceeds only maxSnapshotBytesEstimate (row budget can't bind) and proves completion through the plan-paged reader; a mutant that rethrows snapshot_bytes instead of falling back is killed by exactly this test (plus the store-cap test).

🟡 :555 — collapse inline TTL meta orchestration — PARTIAL, remainder deferred with reasons (a92e14c30): the duplicated session-plan lifecycle (consume-once refresh, offset>0 require-existing, expiry translation) is now one shared createSessionPlanGetter used by both the exact-graph and TTL meta lanes — that's the piece most likely to drift. The full collapse (snapshot + fallback wiring) is deferred: the three lanes differ there for reviewed reasons (SWM data has no snapshot lane and per-call refresh; meta threads its snapshot loader through the same plan getter; exact-graph threads snapshot limits), and folding four behavior fixes and a cross-lane structural rewrite into one PR would obscure the behavioral diff. Slated for the planned graph-plan.ts module split.

🟡 :2264 — single admission owner — DEFERRED with reasons: the invariant (SPARQL discovery is a candidate superset of filterSwmMetaSnapshotRows) is pinned by the set-equivalence regression, which runs both lanes and fails CI on drift; the snapshot path still applies the canonical in-process filter last, so drift degrades to "plan admits more than served", never divergent output. The right fix is the same durable-meta-admission.ts-style extraction, which needs the shared predicate constants moved out of the 3.3k-line file — done properly in the module-split PR rather than as churn here. Happy to prioritize that split next if you want it sooner.

[branarakic]

Canary composition for the fifa-class fixes is up as #1880 (this change carried in full): current testnet-canary + #1868 + #1842, hand-resolved against the #1879 merge, triple-gated green — holdout reconstructs 100/100 quads in ~3s on the exact PR head. Numbers in #1880.