Skip to content

Engine responses over the ws default 100 MiB frame limit tear down the worker connection; state::list has no pagination to bound whole-scope reads #1585

Closed

Description

@7889545

Summary

When a worker reads a state scope whose serialized value exceeds the WebSocket frame limit, the
connection to the engine is torn down, the invocation never returns, and the worker reconnects with a
new worker id — leaving the previous registration behind. This repeats on every path that does a
whole-scope read.

This is the inbound direction (engine → worker). #1579 covers the opposite direction (worker → engine,
16 MiB cap). Both are unbounded-response problems, but they are enforced by different code and need
different fixes.

Observed failure

[iii] WebSocket error: Max payload size exceeded (code=WS_ERR_UNSUPPORTED_MESSAGE_LENGTH)
[iii] WebSocket closed (code=1006)
[iii] Reconnecting in 964ms (attempt 1)...
[iii] Worker registered with ID: e905acc9-b795-4f65-9428-5bc6c86becc5   <- new id, old registration leaked

~3 minutes later the in-flight invocation times out:

[agentmemory] warn Reflect tier failed {"error":"TIMEOUT: invocation timed out after 180000ms"}

The 180 s timeout is collateral damage from our own teardown, not a slow provider: registration at
T+0, timeout at T+171 s, same shape on every occurrence.

Why the frames are large

state::list returns an entire scope as a single RPC response, which becomes a single WS frame (there is
no chunking/streaming in the RPC path). Whole-scope kv.list reads appear in 76 call sites across the
bundled dist/index.mjs (10) and dist/src-*.mjs (66). The heavy ones:

scope serialized size read by
mem:insights 483 MiB insight-list / insight-search / diagnose
mem:graph:edges 133 MiB graph-extract, reflect
mem:graph:nodes 69 MiB graph-extract, reflect
mem:semantic 17 MiB reflect, several searches

Anything over 100 MiB cannot cross the connection at all.

The frame limit is an implicit library default

iii-sdk builds its socket with only two options:

// iii-sdk dist/index.mjs:316
this.ws = new WebSocket(this.address, {
  headers: this.options?.headers,
  handshakeTimeout: WS_HANDSHAKE_TIMEOUT_MS
});

maxPayload is therefore ws's default:

// ws/lib/websocket.js:675
maxPayload: 100 * 1024 * 1024,

Nothing in the project declares or documents this 100 MiB ceiling, so a store that grows past it starts
disconnecting without any code change.

There is no paging primitive to fix this at the call sites

Enumerating the engine registry (iii trigger engine::functions::list, engine 0.22.1) shows the complete
state API:

state::delete  state::get  state::list  state::list_groups  state::set  state::update
  • state::list request schema has exactly one field: scope. No limit / offset / cursor / prefix.
  • state::list_groups returns group (scope) names only — no key counts, no byte sizes, so callers cannot
    cheaply decide "this scope is too big to read".
  • state::get requires a key, and the only way to enumerate keys in a scope is state::list, i.e. the
    whole-scope read itself.

So the "just page the read" fix is not available: there is no bounded read to page with. Newer iii-sdk
(0.24.5) adds local guards (16 MiB outbound error, 64 MiB inbound close), but its changelog states
"Engine limits are unchanged" — the guard makes the failure explicit, it does not make a 483 MiB read
possible.

Why this only started recently (threshold crossing, not a regression)

The same code ran fine for weeks. Measured scope sizes:

scope 2026-09-30 2026-10-17 2026-10-19
graph:edges 10.8 MiB 113.3 MiB 133.2 MiB
graph:nodes 31.8 MiB 59.0 MiB 69.2 MiB
insights — 464.3 MiB 482.9 MiB

graph:edges grew ~6 MiB/day, crossing 100 MiB on 2026-10-16 00:32 local (UTC+8) — which is exactly
when the first disconnect appears in the logs. The previous day had zero disconnects all day, and the
day before that had two (unrelated restarts). Nothing in the code changed at the crossing point.

Before this fix it was also effectively undiagnosable: logError emitted only to the OTel logger, and with
exporter: memory in the observability config the message never reached stdout or a file. Every disconnect
for two days was a bare Reconnecting in Nms (attempt 1) with no reason, no close code and no error text.

Impact

  • 25–29 disconnects per 12 h once graph:edges exceeded the limit — roughly one every 26 minutes. Only
    ~6 of those per 12 h come from the 2-hourly consolidation job; the rest come from the per-session
    graph-extract path.
  • One leaked worker registration per disconnect. Worker ids accumulate for the lifetime of the engine.
  • graph:nodes is at 69 MiB and growing ~5 MiB/day, so it will cross the same limit in roughly a week and
    fragment the same paths again.
  • We have also seen three engine process exits on the same host (2026-10-18 23:20, 2026-10-19 01:47,
    2026-10-19 10:29 local), two of them within ~20 s of a consolidation run that had just produced an
    over-limit read. Causality is not established — there is no engine-side log to check, because the
    in-memory exporter discards it on exit. Flagging the correlation only.

Suggested direction

  1. Shard the large scopes. The repo already ships this idiom: KV.captureEvents(shard) spreads capture
    events over mem:capture:events:00..3f (64 shards) and the read side loops the 64 shards, so each frame
    stays small. Applying it to graph:edges/graph:nodes/insights bounds every frame without touching
    the ~76 call sites if the fan-out lives inside StateKV.list — but it needs a migration for existing
    data, because a half-migrated store is worse than either extreme.
  2. Add pagination to state::list upstream (limit/cursor, or a key-only enumeration + batched gets).
    This is the clean fix and would let every caller stay simple, but it is an engine change.
  3. Declare the limit explicitly in iii-sdk — set maxPayload to a documented value and emit a named
    error including the scope name, instead of surfacing the raw ws code. Even better, log the pending
    invocation + function id when a frame is rejected; the rejected frame never reaches JS, so a receive-side
    size probe cannot see it.

Raising maxPayload alone is not a fix: it moves the memory spike (133 MiB, or 483 MiB for insights) from
the engine into the worker process, which then has to parse and retain it.

Environment

  • agentmemory 0.9.30, iii engine 0.22.1, @iii-dev/helpers 0.22.1, ws 8.21.0
  • Windows, single host, worker and engine on the same machine (loopback)
  • store: ~2.1 GiB across ~2,100 scope files

Also worth mentioning

  • mem::auto-page and the capture pipeline already demonstrate the sharded-scope pattern, so the fix is
    consistent with existing design rather than a new mechanism.
  • state::list_groups would be an ideal place to expose per-scope key counts or byte sizes; that alone
    would let callers degrade gracefully instead of tearing down the connection.

Activity

  1. 7889545 commented on Oct 19, 2026

    @7889545
    Author

    Live reproduction on a 0.9.30 install (Windows), plus a correction to my original attribution

    Two receiver-limit rejections from today's log on the machine that filed this issue. Lines marked [probe] come from a local instrumentation patch of the iii-sdk dist that logs inbound frame sizes >= 8 MiB and dumps in-flight invocations when the receiver rejects a frame; the ws error and close lines are stock.

    12:00:09.900 [probe] Frame probe: inbound 67396394 bytes
    12:00:10.926 [iii] WebSocket error: Max payload size exceeded (code=WS_ERR_UNSUPPORTED_MESSAGE_LENGTH)
    12:00:10.928 [probe] receiver limit rejected an inbound frame; pending invocations (3):
                 mem::graph-extract(waited 17s,id=f8d1bcf1-...) | mem::reflect(waited 3s,...)
    12:00:11.586 [iii] WebSocket closed (code=1006)
    
    13:14:28.189 [probe] Frame probe: inbound 67528462 bytes
    13:14:29.999 [iii] WebSocket error: Max payload size exceeded (code=WS_ERR_UNSUPPORTED_MESSAGE_LENGTH)
    13:14:30.000 [probe] receiver limit rejected an inbound frame; pending invocations (3):
                 mem::graph-extract(waited 21s,id=cf0e136e-...) | mem::reflect(waited 1s,...)
    13:14:33.175 [iii] WebSocket closed (code=1006)
    

    Because the probe only sees frames that parsed, the rejected frame itself is never logged. The largest parsed frames are 17,494,296 B and 67,528,462 B, which line up with mem:semantic (17.1 MiB) and mem:graph:nodes (69.7 MiB) in the store listing below.

    Correction to my original report. I blamed mem::graph-extract. That was wrong. In the 0.9.30 bundle the mem::graph-extract body (dist/index.mjs 12403-12471) contains no kv.list at all, and its persistence path persistGraphDelta (12222-12310) uses point lookups (mem:graph:name-index -> id, mem:graph:edge-key -> id). The whole-scope graph reads that are actually in flight when the frame is rejected are:

    • mem::reflect - a five-way Promise.all including kv.list(KV.graphNodes) and kv.list(KV.graphEdges) (dist/index.mjs 18519-18525), then buildGraphClusters
    • mem::cascade-update - kv.list(KV.graphNodes) (18017) and kv.list(KV.graphEdges) (18039)

    Store sizes (same install, today):

    scope size
    mem:insights 485.6 MiB
    mem:graph:edges 134.0 MiB
    mem:graph:nodes 69.7 MiB
    mem:graph:edge-key 41.2 MiB
    mem:graph:name-index 13.2 MiB
    mem:graph:node-degree 5.7 MiB
    mem:graph:snapshot 3.3 MiB

    mem:graph:edges alone is 34 MiB above the ws default maxPayload (100 MiB) and grows about 6 MiB/day; mem:graph:nodes grows about 5 MiB/day and will cross the same line within roughly a week.

    Link to #1559. That PR (merged 2026-10-16) appears to address exactly these reads - bounded records and point reads for name, character, provenance, adjacency, exact relationship and inventory postings, covering search, graph views, temporal queries, reflection graph context, viewer counts and graph statistics, with src/functions/reflect.ts and src/functions/cascade.ts among the changed files. But the latest release is still v0.9.30 (2026-10-06), so no shipped version contains it. Could a maintainer confirm the release plan for #1559?

    Follow-up gap. #1559's bootstrap path may not be usable on a store like this one: agentmemory graph-index migrate --allow-legacy-scan performs a complete legacy scope scan server-side (the CLI's own error text: 'Initial discovery reads each complete legacy graph scope and can use substantial memory; run it during maintenance'). With mem:graph:edges at 134 MiB, that scan is itself the oversized inbound frame, so the migration can tear the connection down before the indexes that are supposed to remove those reads can be built. A paged or offline discovery path, or documented guidance for stores already above the frame limit, would help. Happy to file that separately if preferred.

  2. 7889545 commented on Oct 19, 2026

    @7889545
    Author

    Correction to my earlier comment — the dominant producer is loadGraphScopes(), not reflect

    My first analysis on this issue named mem::reflect and mem::cascade-update as the producers. That attribution was incomplete. Here is the corrected version, with measurements from a live host (agentmemory 0.9.30, ws 8.21.0, ~167k nodes / ~404k edges).

    1. Patching reflect alone did not stop the teardowns.
    After removing both whole-scope reads from mem::reflect, the same host produced 4 frames of 64.74-64.75 MiB and 3 more rejections inside 12 minutes. reflect was a victim, not the dominant source.

    2. The dominant producer is loadGraphScopes().

    const [nodes, edges] = await Promise.all([this.kv.list(KV.graphNodes), this.kv.list(KV.graphEdges)]);

    It is called by the search graph leg — searchByEntities, expandFromChunks, temporalQuery — so it fires on ordinary retrieval, far more often than reflect. Removing that one read gave a clean 32-minute window: 0 x WS_ERR_UNSUPPORTED_MESSAGE_LENGTH, 0 x close 1006, 52 inbound frames, max 26.73 MiB.

    3. A diagnostic detail that misled me and may mislead others.
    My probe logs a frame only when it is received. A frame rejected at the receiver limit never reaches the message handler, so the offending frame is never printed. The 64.3-64.7 MiB family that immediately precedes every rejection is graph:nodes (73,381,524 B on disk) — accepted and logged. The rejected frame is graph:edges (141,115,592 B = 134.58 MiB), which never appears in the log. Reading the log naively, the 64 MiB frame looks like the culprit; it is not.

    4. state::list is the common denominator.
    Every rejection dumps pending invocations, and state::list is present in all of them. That is the raw RPC behind kv.list — which is why fixing one caller at a time keeps missing the target.

    5. Exposure census (all 232 kv.list call sites).

    scope size on disk whole-scope readers trigger
    mem:insights 487.23 MiB 6 insight-decay-sweep every 24 h automatically; insight-list / insight-search user-facing
    mem:graph:edges 134.58 MiB 11 2 mitigated locally; cascade-update still runs on every memory supersede
    mem:graph:nodes 69.98 MiB 13 under the limit today; ~6 days of growth from crossing it

    6. Correction on reflect, again.
    I earlier reported reflect as failing 100% of the time. That was wrong: the audit trail shows 18 successful reflect runs on the same day. The real pattern is that each attempt first pays the 180 s invocation timeout caused by the torn-down connection, and the retry then completes via the Jaccard fallback. Reflect is degraded, not dead.

    7. What would actually fix this.
    state::list needs a bound: a page/limit parameter, or a server-side response-size guard that fails fast instead of emitting a frame the client is guaranteed to reject. Without it every whole-scope reader is a latent teardown, and the reader list grows with the stores. The per-caller mitigations I applied on this host are damage control — they do not scale, and they cost functionality: I had to disable the graph leg of search and the automatic insight decay sweep to stay connected.

    Related open issues describing the same root: #1450 (adjacency / observation / name-token indexes), #1378 (expandFromChunks blocking), #1360 (unpaginated full-list endpoints), #828 / #825 (per-node edge index).

    Finally, a note on #1559: I confirmed it reworks cascade.ts to use an observation posting index (createGraphIndexReader + reader.posting("observation", ...)), which addresses the cascade path properly. But it is merged and unreleased (latest release is still v0.9.30), and its migration reads complete legacy graph scopes server-side — the same wall, since mem:graph:edges at 134.58 MiB already exceeds the inbound frame limit on exactly the stores that need the index.

  3. yoropico commented on Oct 21, 2026

    @yoropico

    Same failure here, with numbers from a 3-month self-hosted store (one instance shared by ~15 Claude Code sessions plus a PM harness), in case they help size the fix:

    • Scopes as stored by 0.9.29 / iii 0.11.2: mem:memories 168 MB (34,631 memories), mem:graph:nodes 200 MB (48,690 nodes, 47,152 of them file nodes), mem:graph:edges 59 MB, mem:audit 223 MB, observations 514 MB over 6,553 per-session keys, live search index 556 MB.
    • On 0.9.29 we only stay up because the iii-sdk socket is built with maxPayload: 1 GiB (otherwise kv.list(mem:graph:nodes) tears the worker down exactly as described).
    • On pristine 0.9.30 / iii 0.22.1 against a copy of that store: /agentmemory/counts reports memories: 0, /memories answers Function api::memories not found, /search answers invocation_stopped, and the worker re-registers ("[iii] Reconnecting ... attempt 1") after the first whole-scope read. Raising the sdk maxPayload to 1 GiB on 0.22.1 (channels-*.mjs/cjs) does not help any more; iii-worker-manager 0.22.1 rejects a max_message_size key (unknown field), and iii 0.25's fragmented messages top out at 64 MiB, so the memories scope still does not fit.
    • Boot of that store takes 9-20 s (file-based engine load), so 0.9.30 exits at every start on a moderately large store: hardcoded 15 s worker readiness wait #1530/fix: allow slow workers to finish startup across CLI paths #1531 is needed as well; with the wait lifted the worker is ready at ~12-15 s.

    So from this side the only durable path looks like paged/keyed reads for the large scopes (#1450 / #893 direction) rather than a bigger frame.

  4. 7889545 commented on Oct 25, 2026

    @7889545
    Author

    Three more whole-scope readers, plus a heads-up: the iii-sdk 0.25.0 upgrade lowers the receive cap from 100 MiB to 64 MiB.

    1. Three unbounded mem:insights reads in src/functions/reflect.ts (main)

    All three do kv.list<Insight>(KV.insights) and filter in JS afterwards, so the state::list response carries the whole scope:

    registered function code what it does
    mem::insight-list let items = await kv.list<Insight>(KV.insights); then .filter(...) default limit 50, but the read is unbounded
    mem::insight-search same read, then filter on data.query.toLowerCase() default limit 10, same unbounded read
    mem::insight-decay-sweep const items = await kv.list<Insight>(KV.insights); runs every 86400000 ms, gated only by INSIGHT_DECAY_ENABLED !== "false"

    The sweep is the one that matters most: it fires on a timer with no user action, so a host that never calls an insight function still pays the read.

    Store size on the host that filed this, measured from data/state_store.db:

    mem%3Ainsights.bin    522.8 MB   2026-10-25   (was 487.2 MB on 2026-10-19)
    mem%3Agraph%3Aedges.bin 146.5 MB  2026-10-25
    mem%3Agraph%3Anodes.bin  76.2 MB  2026-10-25
    

    So a single state::list(mem:insights) response is ~523 MB today and grew ~35 MB in six days on this host.

    2. Heads-up on iii-sdk 0.25.0 (PR #1592): the receive cap goes DOWN

    iii-sdk@0.25.0 introduces explicit limits:

    const MAX_JSON_FRAME_BYTES    = 16 * 1024 * 1024;   // 16 MiB
    const MAX_JSON_FRAGMENT_BYTES = 1024 * 1024;        // 1 MiB
    const MAX_JSON_MESSAGE_BYTES  = 64 * 1024 * 1024;   // 64 MiB
    
    this.ws = new WebSocket(this.address, { maxPayload: MAX_JSON_MESSAGE_BYTES, ... });

    Two genuinely good changes there:

    • sendMessageRaw now fragments anything above 16 MiB into 1 MiB WebSocket continuation frames (fin: false on all but the last), so a large outgoing message no longer arrives as one giant frame.
    • prepareJson turns an oversized envelope into a catchable payload_too_large InvocationError (or an error result when it is an invocationresult) instead of tearing the connection down.

    But note the direction: maxPayload is the receive limit, and 64 MiB is stricter than the ws default of 100 MiB that 0.22.1 was relying on (0.22.1 sets no maxPayload at all — verified against the published package). The engine side is Rust, so the JS-side guard does not necessarily apply to what the engine sends back.

    On our host, the frames that are currently accepted reach 64.75 MiB — already above 64 MiB. The rejected one was mem:graph:edges at 141,115,592 B (134.58 MiB). So after the 0.25.0 upgrade the same traffic meets the cap roughly a third earlier than before, and the failure mode for the inbound direction is unchanged.

    That is not an argument against #1592 — the fragmentation and the graceful error are real improvements. It is an argument that bounding the reads (or paginating state::list) is still the fix, because the upgrade alone moves the threshold the wrong way for stores like this one.

    Happy to re-measure on 0.9.31 + iii 0.25.0 once it ships, if that is useful.

  5. aberenyi commented on Oct 27, 2026

    @aberenyi

    Data point from 0.9.31 (iii 0.25.0, macOS arm64, file state backend), in case it helps scope the pagination work.

    On iii 0.25.0 the worker no longer drops when a whole-scope read is too large. The call fails cleanly instead:

    payload_too_large: Serialized JSON envelope is 30224696 bytes; limit is 16777216 bytes. Use channels for large data.

    Our store at the time: mem:semantic was 30 MB (25,153 facts) and mem:insights was 369 MB (5,935 records). GET /semantic and GET /insights both returned that error. The worker stayed up (its uptime did not reset), and unscoped memory_diagnose and memory_insight_list over MCP returned Internal error. So on 0.25 a scope over 16 MiB no longer takes the daemon down, but every call that lists that scope is dead.

    Two things that may matter for the design:

    1. Paging after the list cannot fix it. I backported fix(api): page /semantic and /procedural so they fit the frame limit #1447 (list, then sort and slice) into the 0.9.31 bundle. GET /semantic?limit=3 still failed with the error above, because kv.list(KV.semantic) itself is over the limit. Paging has to happen in state::list, or as a key-only listing plus batched gets, as proposed here.

    2. Background jobs read the semantic scope in full, not only the endpoint. In the 0.9.31 bundle, mem::consolidate-pipeline calls kv.list(KV.semantic) on every run (every 2 hours) only to look for a duplicate fact with find(). A grep of the bundle also shows mem::reflect, mem::retention-score, api::compression-retry, mem::import and the mesh functions listing the same scope (attributed by nearest registerFunction, so approximate). I have not yet seen the consolidation run fail after the upgrade, because the first run since the restart is still due. A keyed lookup or an index for that dedupe would remove the largest caller.

    For anyone who hits this before pagination lands: 62% of our mem:semantic bytes were sourceSessionIds (the write described in #1360). Trimming that array to one id on the stopped store took the scope from 30.9 MB to 12.7 MB, and GET /semantic returned 200 again with total 25153. At about 266 new facts a day that is roughly 12 days of headroom, so it is a stopgap, not a fix.

  6. 7889545 commented on Oct 28, 2026

    @7889545
    Author

    Adding measurements from agentmemory 0.9.31 / iii-sdk 0.25.0 that may help scope the fix. Everything below is read from the published npm tarball and the v0.9.31 source tarball. Each count names its base file and match expression, because summing across the 13 bundles in dist/ double-counts: both index.mjs (1,162,189 B) and src-DaMtDzt_.mjs (864,450 B) embed a full copy of the module layer, so kv.list(KV.insights appears 5 times in each and only becomes 10 if you add them.

    1. The app-side guard has 5 call sites, and none of them is a whole-scope reader

    src/state/frame-guard.ts defines checkPayloadFrameSize with SAFE_PAYLOAD_BYTES = 15 * 1024 * 1024. The only importers:

    file imported
    src/functions/compression-recovery.ts checkPayloadFrameSize
    src/functions/export-import.ts checkPayloadFrameSize, payloadByteLength, SAFE_PAYLOAD_BYTES
    src/functions/observation-source-budget.ts same
    src/functions/replay.ts checkPayloadFrameSize
    src/triggers/api.ts checkPayloadFrameSize

    dist cross-check, base dist/index.mjs only: checkPayloadFrameSize( = 6 (1 definition + 5 calls), SAFE_PAYLOAD_BYTES = 5.

    2. state/kv.ts still has a single read entry point with no cursor

    async list<T = unknown>(scope: string): Promise<T[]>

    Counting kv.list( by scope constant across the 254 .ts files in src/ gives 251 calls. Largest groups: KV.sessions 31, KV.memories 28, KV.observations 19, KV.actions 17, KV.actionEdges 15, KV.semantic 10, KV.procedural 9, KV.leases 9, KV.lessons 8, KV.facets 8, KV.summaries 7, KV.crystals 5, KV.insights 5.

    The graph scopes are the one exception on the retrieval side: graph-retrieval.ts and graph-retrieval-index.ts contain zero kv.list (they go through createGraphIndexReader(kv, { maxReads: 512 }) in cascade.ts:48), although functions/graph.ts still has 5 real kv.list calls on the rebuild / legacy-scan path.

    3. The cursor paging added in 0.9.31 bounds the response, not the read

    src/state/list-query.ts (113 lines) exports LIST_PAGE_MAX = 500, parseListQuery, encodeCursor/decodeCursor, sortByKeyDesc, pageAfterCursor, pageByOffset; it is used by mcp/session-arguments.ts, mcp/standalone.ts, triggers/api.ts (4 sites) and functions/actions.ts. But the shape gives it away:

    export function pageAfterCursor<T>(
      sortedDesc: T[],   // already-materialized full array
      ...
      const page = sortedDesc.slice(start, start + limit);

    and mcp/standalone.ts:423:

    const sessions = (await kvInstance.list<Session>("mem:sessions"))
      .filter(...).filter(...).filter(...);
    const { page, nextCursor } = pageAfterCursor(sortByKeyDesc(sessions, ...), ...);

    The whole scope is still materialized by kv.list before paging happens, so this bounds the worker-to-client frame but not the engine-to-worker RPC response, which is the frame that trips the limit.

    4. The 16 MiB limit is not in the app bundle

    Base dist/index.mjs only: payload_too_large = 0, 16777216 = 0, MAX_JSON_FRAME_BYTES = 0. So the envelope guard lives in iii-sdk 0.25.0 and no app-level change can raise or bypass it; the app's own 15 MiB check is a separate, narrower safety line.

    5. Size distribution on our host, for scoping

    A local 0.9.30 store: 2,158 flat .bin files, 2,267 MiB total, exactly 7 scopes above 16 MiB: mem:insights 523.67 MiB, mem:graph:edges 146.55, mem:graph:nodes 76.23, mem:graph:edge-key 44.99, two mem:obs:session-* at 27.00 and 23.15, and mem:semantic 19.40. The next largest is 14.37, so nothing sits in the 15 to 16 MiB band. On 0.9.30 one mem:insights read emits roughly the whole scope in a single frame and tears down the worker; on 0.9.31 the same read fails with payload_too_large and the worker survives, matching @aberenyi's 30,224,696 vs 16,777,216 report.

    What would actually fix it. Agreeing with @aberenyi: paging has to be in state::list. A bounded read (limit plus continuation token) on the engine side would let a caller walk a 523 MiB scope without any single frame approaching the limit; response-side cursor paging cannot help, because the oversized frame is emitted before paging begins. If a branch build is available, I can test it against the 523 MiB mem:insights scope on Windows.

  7. Levako commented on Nov 4, 2026

    @Levako

    Data point from 0.9.31 (iii 0.25.0, Linux, file state backend, systemd), plus the cause of the mem:insights bloat in our store.

    Envelope sizes on 0.9.31. iii trigger -n agentmemory-3111 state::list scope=<scope>:

    scope envelope result
    mem:lessons < 16 MiB ok
    mem:semantic (~25.9k facts) 31,778,615 B payload_too_large
    mem:graph:nodes 20,207,859 B payload_too_large
    mem:graph:edges 64,841,738 B payload_too_large
    mem:insights (28,798 records) 288,728,554 B payload_too_large

    Anything that lists those scopes fails. That includes mem::reflect (semantic + insights) and mem::insight-decay-sweep. Graph enrichment reported unavailable, and the graph index stayed at needs-migration. We rolled back to 0.9.30 the same day.

    Why mem:insights is so large. We parsed the scope file. 266.4 MB of its 299.1 MB of JSON (89%) is sourceMemoryIds. In 0.9.30, mem::reflect stores sourceMemoryIds: cluster.factIds, which is every semantic fact in the concept cluster, not the facts the insight cites. Clusters grow with the semantic store, so the arrays grow too. Median ids per insight by creation month: Jul 50, Aug 268, Sep 116, Oct 268, Nov 452. The max is 3,221 (about 93 KB for one record).

    0.9.31's parseInsight keeps only the sources the LLM cites (F<n> labels), which fixes new writes. Existing records are not migrated, though, so an upgraded store keeps the bloat. That probably explains the very large insights scopes reported above, such as 369 MB for 5,935 records (~62 KB per record).

    Mitigation, and why it isn't enough. We capped existing records at 10 ids on 0.9.30. Keys came from the on-disk scope file, since state::list can't return the scope. Each oversized key then got one state::update with [{type:"set", path:"sourceMemoryIds", value: ids.slice(0, 10)}], which leaves the other fields alone under concurrent reinforce/decay writes.

    • 25,161 records updated in 19 s, 0 failures; content, title and tags unchanged against a snapshot.
    • The scope file went from 289.4 MB to 39.7 MB.

    That is still over 16 MiB, and mem:semantic (31.8 MB) and mem:graph:edges (64.8 MB) can't shrink without deleting data. So this only helps; paging in state::list (or key-only listing plus batched gets) is still required. A one-time migration that caps sourceMemoryIds on pre-0.9.31 insights would be a cheap companion fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions