Repository navigation
Description
Summary
When a worker reads a state scope whose serialized value exceeds the WebSocket frame limit, the
connection to the engine is torn down, the invocation never returns, and the worker reconnects with a
new worker id — leaving the previous registration behind. This repeats on every path that does a
whole-scope read.
This is the inbound direction (engine → worker). #1579 covers the opposite direction (worker → engine,
16 MiB cap). Both are unbounded-response problems, but they are enforced by different code and need
different fixes.
Observed failure
[iii] WebSocket error: Max payload size exceeded (code=WS_ERR_UNSUPPORTED_MESSAGE_LENGTH)
[iii] WebSocket closed (code=1006)
[iii] Reconnecting in 964ms (attempt 1)...
[iii] Worker registered with ID: e905acc9-b795-4f65-9428-5bc6c86becc5 <- new id, old registration leaked
~3 minutes later the in-flight invocation times out:
[agentmemory] warn Reflect tier failed {"error":"TIMEOUT: invocation timed out after 180000ms"}
The 180 s timeout is collateral damage from our own teardown, not a slow provider: registration at
T+0, timeout at T+171 s, same shape on every occurrence.
Why the frames are large
state::list returns an entire scope as a single RPC response, which becomes a single WS frame (there is
no chunking/streaming in the RPC path). Whole-scope kv.list reads appear in 76 call sites across the
bundled dist/index.mjs (10) and dist/src-*.mjs (66). The heavy ones:
| scope | serialized size | read by |
|---|---|---|
mem:insights |
483 MiB | insight-list / insight-search / diagnose |
mem:graph:edges |
133 MiB | graph-extract, reflect |
mem:graph:nodes |
69 MiB | graph-extract, reflect |
mem:semantic |
17 MiB | reflect, several searches |
Anything over 100 MiB cannot cross the connection at all.
The frame limit is an implicit library default
iii-sdk builds its socket with only two options:
// iii-sdk dist/index.mjs:316
this.ws = new WebSocket(this.address, {
headers: this.options?.headers,
handshakeTimeout: WS_HANDSHAKE_TIMEOUT_MS
});maxPayload is therefore ws's default:
// ws/lib/websocket.js:675
maxPayload: 100 * 1024 * 1024,Nothing in the project declares or documents this 100 MiB ceiling, so a store that grows past it starts
disconnecting without any code change.
There is no paging primitive to fix this at the call sites
Enumerating the engine registry (iii trigger engine::functions::list, engine 0.22.1) shows the complete
state API:
state::delete state::get state::list state::list_groups state::set state::update
state::listrequest schema has exactly one field:scope. Nolimit/offset/cursor/prefix.state::list_groupsreturns group (scope) names only — no key counts, no byte sizes, so callers cannot
cheaply decide "this scope is too big to read".state::getrequires a key, and the only way to enumerate keys in a scope isstate::list, i.e. the
whole-scope read itself.
So the "just page the read" fix is not available: there is no bounded read to page with. Newer iii-sdk
(0.24.5) adds local guards (16 MiB outbound error, 64 MiB inbound close), but its changelog states
"Engine limits are unchanged" — the guard makes the failure explicit, it does not make a 483 MiB read
possible.
Why this only started recently (threshold crossing, not a regression)
The same code ran fine for weeks. Measured scope sizes:
| scope | 2026-09-30 | 2026-10-17 | 2026-10-19 |
|---|---|---|---|
graph:edges |
10.8 MiB | 113.3 MiB | 133.2 MiB |
graph:nodes |
31.8 MiB | 59.0 MiB | 69.2 MiB |
insights |
— | 464.3 MiB | 482.9 MiB |
graph:edges grew ~6 MiB/day, crossing 100 MiB on 2026-10-16 00:32 local (UTC+8) — which is exactly
when the first disconnect appears in the logs. The previous day had zero disconnects all day, and the
day before that had two (unrelated restarts). Nothing in the code changed at the crossing point.
Before this fix it was also effectively undiagnosable: logError emitted only to the OTel logger, and with
exporter: memory in the observability config the message never reached stdout or a file. Every disconnect
for two days was a bare Reconnecting in Nms (attempt 1) with no reason, no close code and no error text.
Impact
- 25–29 disconnects per 12 h once
graph:edgesexceeded the limit — roughly one every 26 minutes. Only
~6 of those per 12 h come from the 2-hourly consolidation job; the rest come from the per-session
graph-extractpath. - One leaked worker registration per disconnect. Worker ids accumulate for the lifetime of the engine.
graph:nodesis at 69 MiB and growing ~5 MiB/day, so it will cross the same limit in roughly a week and
fragment the same paths again.- We have also seen three engine process exits on the same host (2026-10-18 23:20, 2026-10-19 01:47,
2026-10-19 10:29 local), two of them within ~20 s of a consolidation run that had just produced an
over-limit read. Causality is not established — there is no engine-side log to check, because the
in-memory exporter discards it on exit. Flagging the correlation only.
Suggested direction
- Shard the large scopes. The repo already ships this idiom:
KV.captureEvents(shard)spreads capture
events overmem:capture:events:00..3f(64 shards) and the read side loops the 64 shards, so each frame
stays small. Applying it tograph:edges/graph:nodes/insightsbounds every frame without touching
the ~76 call sites if the fan-out lives insideStateKV.list— but it needs a migration for existing
data, because a half-migrated store is worse than either extreme. - Add pagination to
state::listupstream (limit/cursor, or a key-only enumeration + batched gets).
This is the clean fix and would let every caller stay simple, but it is an engine change. - Declare the limit explicitly in
iii-sdk— setmaxPayloadto a documented value and emit a named
error including the scope name, instead of surfacing the rawwscode. Even better, log the pending
invocation + function id when a frame is rejected; the rejected frame never reaches JS, so a receive-side
size probe cannot see it.
Raising maxPayload alone is not a fix: it moves the memory spike (133 MiB, or 483 MiB for insights) from
the engine into the worker process, which then has to parse and retain it.
Environment
- agentmemory 0.9.30, iii engine 0.22.1, @iii-dev/helpers 0.22.1, ws 8.21.0
- Windows, single host, worker and engine on the same machine (loopback)
- store: ~2.1 GiB across ~2,100 scope files
Also worth mentioning
mem::auto-pageand the capture pipeline already demonstrate the sharded-scope pattern, so the fix is
consistent with existing design rather than a new mechanism.state::list_groupswould be an ideal place to expose per-scope key counts or byte sizes; that alone
would let callers degrade gracefully instead of tearing down the connection.
Activity
7889545 commentedon Oct 19, 2026
Live reproduction on a 0.9.30 install (Windows), plus a correction to my original attribution
Two receiver-limit rejections from today's log on the machine that filed this issue. Lines marked
[probe]come from a local instrumentation patch of the iii-sdk dist that logs inbound frame sizes >= 8 MiB and dumps in-flight invocations when the receiver rejects a frame; thewserror and close lines are stock.Because the probe only sees frames that parsed, the rejected frame itself is never logged. The largest parsed frames are 17,494,296 B and 67,528,462 B, which line up with
mem:semantic(17.1 MiB) andmem:graph:nodes(69.7 MiB) in the store listing below.Correction to my original report. I blamed
mem::graph-extract. That was wrong. In the 0.9.30 bundle themem::graph-extractbody (dist/index.mjs 12403-12471) contains nokv.listat all, and its persistence pathpersistGraphDelta(12222-12310) uses point lookups (mem:graph:name-index-> id,mem:graph:edge-key-> id). The whole-scope graph reads that are actually in flight when the frame is rejected are:mem::reflect- a five-wayPromise.allincludingkv.list(KV.graphNodes)andkv.list(KV.graphEdges)(dist/index.mjs 18519-18525), thenbuildGraphClustersmem::cascade-update-kv.list(KV.graphNodes)(18017) andkv.list(KV.graphEdges)(18039)Store sizes (same install, today):
mem:graph:edgesalone is 34 MiB above thewsdefaultmaxPayload(100 MiB) and grows about 6 MiB/day;mem:graph:nodesgrows about 5 MiB/day and will cross the same line within roughly a week.Link to #1559. That PR (merged 2026-10-16) appears to address exactly these reads - bounded records and point reads for name, character, provenance, adjacency, exact relationship and inventory postings, covering search, graph views, temporal queries, reflection graph context, viewer counts and graph statistics, with
src/functions/reflect.tsandsrc/functions/cascade.tsamong the changed files. But the latest release is still v0.9.30 (2026-10-06), so no shipped version contains it. Could a maintainer confirm the release plan for #1559?Follow-up gap. #1559's bootstrap path may not be usable on a store like this one:
agentmemory graph-index migrate --allow-legacy-scanperforms a complete legacy scope scan server-side (the CLI's own error text: 'Initial discovery reads each complete legacy graph scope and can use substantial memory; run it during maintenance'). Withmem:graph:edgesat 134 MiB, that scan is itself the oversized inbound frame, so the migration can tear the connection down before the indexes that are supposed to remove those reads can be built. A paged or offline discovery path, or documented guidance for stores already above the frame limit, would help. Happy to file that separately if preferred.7889545 commentedon Oct 19, 2026
Correction to my earlier comment — the dominant producer is
loadGraphScopes(), notreflectMy first analysis on this issue named
mem::reflectandmem::cascade-updateas the producers. That attribution was incomplete. Here is the corrected version, with measurements from a live host (agentmemory 0.9.30, ws 8.21.0, ~167k nodes / ~404k edges).1. Patching
reflectalone did not stop the teardowns.After removing both whole-scope reads from
mem::reflect, the same host produced 4 frames of 64.74-64.75 MiB and 3 more rejections inside 12 minutes.reflectwas a victim, not the dominant source.2. The dominant producer is
loadGraphScopes().It is called by the search graph leg —
searchByEntities,expandFromChunks,temporalQuery— so it fires on ordinary retrieval, far more often thanreflect. Removing that one read gave a clean 32-minute window: 0 xWS_ERR_UNSUPPORTED_MESSAGE_LENGTH, 0 x close 1006, 52 inbound frames, max 26.73 MiB.3. A diagnostic detail that misled me and may mislead others.
My probe logs a frame only when it is received. A frame rejected at the receiver limit never reaches the message handler, so the offending frame is never printed. The 64.3-64.7 MiB family that immediately precedes every rejection is
graph:nodes(73,381,524 B on disk) — accepted and logged. The rejected frame isgraph:edges(141,115,592 B = 134.58 MiB), which never appears in the log. Reading the log naively, the 64 MiB frame looks like the culprit; it is not.4.
state::listis the common denominator.Every rejection dumps pending invocations, and
state::listis present in all of them. That is the raw RPC behindkv.list— which is why fixing one caller at a time keeps missing the target.5. Exposure census (all 232
kv.listcall sites).mem:insightsinsight-decay-sweepevery 24 h automatically;insight-list/insight-searchuser-facingmem:graph:edgescascade-updatestill runs on every memory supersedemem:graph:nodes6. Correction on reflect, again.
I earlier reported reflect as failing 100% of the time. That was wrong: the audit trail shows 18 successful reflect runs on the same day. The real pattern is that each attempt first pays the 180 s invocation timeout caused by the torn-down connection, and the retry then completes via the Jaccard fallback. Reflect is degraded, not dead.
7. What would actually fix this.
state::listneeds a bound: a page/limit parameter, or a server-side response-size guard that fails fast instead of emitting a frame the client is guaranteed to reject. Without it every whole-scope reader is a latent teardown, and the reader list grows with the stores. The per-caller mitigations I applied on this host are damage control — they do not scale, and they cost functionality: I had to disable the graph leg of search and the automatic insight decay sweep to stay connected.Related open issues describing the same root: #1450 (adjacency / observation / name-token indexes), #1378 (
expandFromChunksblocking), #1360 (unpaginated full-list endpoints), #828 / #825 (per-node edge index).Finally, a note on #1559: I confirmed it reworks
cascade.tsto use an observation posting index (createGraphIndexReader+reader.posting("observation", ...)), which addresses the cascade path properly. But it is merged and unreleased (latest release is still v0.9.30), and its migration reads complete legacy graph scopes server-side — the same wall, sincemem:graph:edgesat 134.58 MiB already exceeds the inbound frame limit on exactly the stores that need the index.yoropico commentedon Oct 21, 2026
Same failure here, with numbers from a 3-month self-hosted store (one instance shared by ~15 Claude Code sessions plus a PM harness), in case they help size the fix:
mem:memories168 MB (34,631 memories),mem:graph:nodes200 MB (48,690 nodes, 47,152 of them file nodes),mem:graph:edges59 MB,mem:audit223 MB, observations 514 MB over 6,553 per-session keys, live search index 556 MB.maxPayload: 1 GiB(otherwisekv.list(mem:graph:nodes)tears the worker down exactly as described)./agentmemory/countsreportsmemories: 0,/memoriesanswersFunction api::memories not found,/searchanswersinvocation_stopped, and the worker re-registers ("[iii] Reconnecting ... attempt 1") after the first whole-scope read. Raising the sdkmaxPayloadto 1 GiB on 0.22.1 (channels-*.mjs/cjs) does not help any more;iii-worker-manager0.22.1 rejects amax_message_sizekey (unknown field), and iii 0.25's fragmented messages top out at 64 MiB, so the memories scope still does not fit.So from this side the only durable path looks like paged/keyed reads for the large scopes (#1450 / #893 direction) rather than a bigger frame.
7889545 commentedon Oct 25, 2026
Three more whole-scope readers, plus a heads-up: the iii-sdk 0.25.0 upgrade lowers the receive cap from 100 MiB to 64 MiB.
1. Three unbounded
mem:insightsreads insrc/functions/reflect.ts(main)All three do
kv.list<Insight>(KV.insights)and filter in JS afterwards, so thestate::listresponse carries the whole scope:mem::insight-listlet items = await kv.list<Insight>(KV.insights);then.filter(...)limit50, but the read is unboundedmem::insight-searchdata.query.toLowerCase()limit10, same unbounded readmem::insight-decay-sweepconst items = await kv.list<Insight>(KV.insights);86400000ms, gated only byINSIGHT_DECAY_ENABLED !== "false"The sweep is the one that matters most: it fires on a timer with no user action, so a host that never calls an insight function still pays the read.
Store size on the host that filed this, measured from
data/state_store.db:So a single
state::list(mem:insights)response is ~523 MB today and grew ~35 MB in six days on this host.2. Heads-up on iii-sdk 0.25.0 (PR #1592): the receive cap goes DOWN
iii-sdk@0.25.0introduces explicit limits:Two genuinely good changes there:
sendMessageRawnow fragments anything above 16 MiB into 1 MiB WebSocket continuation frames (fin: falseon all but the last), so a large outgoing message no longer arrives as one giant frame.prepareJsonturns an oversized envelope into a catchablepayload_too_largeInvocationError(or an error result when it is aninvocationresult) instead of tearing the connection down.But note the direction:
maxPayloadis the receive limit, and 64 MiB is stricter than thewsdefault of 100 MiB that 0.22.1 was relying on (0.22.1 sets nomaxPayloadat all — verified against the published package). The engine side is Rust, so the JS-side guard does not necessarily apply to what the engine sends back.On our host, the frames that are currently accepted reach 64.75 MiB — already above 64 MiB. The rejected one was
mem:graph:edgesat 141,115,592 B (134.58 MiB). So after the 0.25.0 upgrade the same traffic meets the cap roughly a third earlier than before, and the failure mode for the inbound direction is unchanged.That is not an argument against #1592 — the fragmentation and the graceful error are real improvements. It is an argument that bounding the reads (or paginating
state::list) is still the fix, because the upgrade alone moves the threshold the wrong way for stores like this one.Happy to re-measure on 0.9.31 + iii 0.25.0 once it ships, if that is useful.
aberenyi commentedon Oct 27, 2026
Data point from 0.9.31 (iii 0.25.0, macOS arm64, file state backend), in case it helps scope the pagination work.
On iii 0.25.0 the worker no longer drops when a whole-scope read is too large. The call fails cleanly instead:
payload_too_large: Serialized JSON envelope is 30224696 bytes; limit is 16777216 bytes. Use channels for large data.Our store at the time:
mem:semanticwas 30 MB (25,153 facts) andmem:insightswas 369 MB (5,935 records).GET /semanticandGET /insightsboth returned that error. The worker stayed up (its uptime did not reset), and unscopedmemory_diagnoseandmemory_insight_listover MCP returnedInternal error. So on 0.25 a scope over 16 MiB no longer takes the daemon down, but every call that lists that scope is dead.Two things that may matter for the design:
Paging after the list cannot fix it. I backported fix(api): page /semantic and /procedural so they fit the frame limit #1447 (list, then sort and slice) into the 0.9.31 bundle.
GET /semantic?limit=3still failed with the error above, becausekv.list(KV.semantic)itself is over the limit. Paging has to happen instate::list, or as a key-only listing plus batched gets, as proposed here.Background jobs read the semantic scope in full, not only the endpoint. In the 0.9.31 bundle,
mem::consolidate-pipelinecallskv.list(KV.semantic)on every run (every 2 hours) only to look for a duplicate fact withfind(). A grep of the bundle also showsmem::reflect,mem::retention-score,api::compression-retry,mem::importand the mesh functions listing the same scope (attributed by nearestregisterFunction, so approximate). I have not yet seen the consolidation run fail after the upgrade, because the first run since the restart is still due. A keyed lookup or an index for that dedupe would remove the largest caller.For anyone who hits this before pagination lands: 62% of our
mem:semanticbytes weresourceSessionIds(the write described in #1360). Trimming that array to one id on the stopped store took the scope from 30.9 MB to 12.7 MB, andGET /semanticreturned 200 again withtotal25153. At about 266 new facts a day that is roughly 12 days of headroom, so it is a stopgap, not a fix.7889545 commentedon Oct 28, 2026
Adding measurements from agentmemory 0.9.31 / iii-sdk 0.25.0 that may help scope the fix. Everything below is read from the published npm tarball and the v0.9.31 source tarball. Each count names its base file and match expression, because summing across the 13 bundles in
dist/double-counts: bothindex.mjs(1,162,189 B) andsrc-DaMtDzt_.mjs(864,450 B) embed a full copy of the module layer, sokv.list(KV.insightsappears 5 times in each and only becomes 10 if you add them.1. The app-side guard has 5 call sites, and none of them is a whole-scope reader
src/state/frame-guard.tsdefinescheckPayloadFrameSizewithSAFE_PAYLOAD_BYTES = 15 * 1024 * 1024. The only importers:src/functions/compression-recovery.tscheckPayloadFrameSizesrc/functions/export-import.tscheckPayloadFrameSize,payloadByteLength,SAFE_PAYLOAD_BYTESsrc/functions/observation-source-budget.tssrc/functions/replay.tscheckPayloadFrameSizesrc/triggers/api.tscheckPayloadFrameSizedist cross-check, base
dist/index.mjsonly:checkPayloadFrameSize(= 6 (1 definition + 5 calls),SAFE_PAYLOAD_BYTES= 5.2.
state/kv.tsstill has a single read entry point with no cursorCounting
kv.list(by scope constant across the 254.tsfiles insrc/gives 251 calls. Largest groups:KV.sessions31,KV.memories28,KV.observations19,KV.actions17,KV.actionEdges15,KV.semantic10,KV.procedural9,KV.leases9,KV.lessons8,KV.facets8,KV.summaries7,KV.crystals5,KV.insights5.The graph scopes are the one exception on the retrieval side:
graph-retrieval.tsandgraph-retrieval-index.tscontain zerokv.list(they go throughcreateGraphIndexReader(kv, { maxReads: 512 })incascade.ts:48), althoughfunctions/graph.tsstill has 5 realkv.listcalls on the rebuild / legacy-scan path.3. The cursor paging added in 0.9.31 bounds the response, not the read
src/state/list-query.ts(113 lines) exportsLIST_PAGE_MAX = 500,parseListQuery,encodeCursor/decodeCursor,sortByKeyDesc,pageAfterCursor,pageByOffset; it is used bymcp/session-arguments.ts,mcp/standalone.ts,triggers/api.ts(4 sites) andfunctions/actions.ts. But the shape gives it away:and
mcp/standalone.ts:423:The whole scope is still materialized by
kv.listbefore paging happens, so this bounds the worker-to-client frame but not the engine-to-worker RPC response, which is the frame that trips the limit.4. The 16 MiB limit is not in the app bundle
Base
dist/index.mjsonly:payload_too_large= 0,16777216= 0,MAX_JSON_FRAME_BYTES= 0. So the envelope guard lives in iii-sdk 0.25.0 and no app-level change can raise or bypass it; the app's own 15 MiB check is a separate, narrower safety line.5. Size distribution on our host, for scoping
A local 0.9.30 store: 2,158 flat
.binfiles, 2,267 MiB total, exactly 7 scopes above 16 MiB:mem:insights523.67 MiB,mem:graph:edges146.55,mem:graph:nodes76.23,mem:graph:edge-key44.99, twomem:obs:session-*at 27.00 and 23.15, andmem:semantic19.40. The next largest is 14.37, so nothing sits in the 15 to 16 MiB band. On 0.9.30 onemem:insightsread emits roughly the whole scope in a single frame and tears down the worker; on 0.9.31 the same read fails withpayload_too_largeand the worker survives, matching @aberenyi's 30,224,696 vs 16,777,216 report.What would actually fix it. Agreeing with @aberenyi: paging has to be in
state::list. A bounded read (limit plus continuation token) on the engine side would let a caller walk a 523 MiB scope without any single frame approaching the limit; response-side cursor paging cannot help, because the oversized frame is emitted before paging begins. If a branch build is available, I can test it against the 523 MiBmem:insightsscope on Windows.Levako commentedon Nov 4, 2026
Data point from 0.9.31 (iii 0.25.0, Linux, file state backend, systemd), plus the cause of the
mem:insightsbloat in our store.Envelope sizes on 0.9.31.
iii trigger -n agentmemory-3111 state::list scope=<scope>:mem:lessonsmem:semantic(~25.9k facts)payload_too_largemem:graph:nodespayload_too_largemem:graph:edgespayload_too_largemem:insights(28,798 records)payload_too_largeAnything that lists those scopes fails. That includes
mem::reflect(semantic + insights) andmem::insight-decay-sweep. Graph enrichment reported unavailable, and the graph index stayed atneeds-migration. We rolled back to 0.9.30 the same day.Why
mem:insightsis so large. We parsed the scope file. 266.4 MB of its 299.1 MB of JSON (89%) issourceMemoryIds. In 0.9.30,mem::reflectstoressourceMemoryIds: cluster.factIds, which is every semantic fact in the concept cluster, not the facts the insight cites. Clusters grow with the semantic store, so the arrays grow too. Median ids per insight by creation month: Jul 50, Aug 268, Sep 116, Oct 268, Nov 452. The max is 3,221 (about 93 KB for one record).0.9.31's
parseInsightkeeps only the sources the LLM cites (F<n>labels), which fixes new writes. Existing records are not migrated, though, so an upgraded store keeps the bloat. That probably explains the very large insights scopes reported above, such as 369 MB for 5,935 records (~62 KB per record).Mitigation, and why it isn't enough. We capped existing records at 10 ids on 0.9.30. Keys came from the on-disk scope file, since
state::listcan't return the scope. Each oversized key then got onestate::updatewith[{type:"set", path:"sourceMemoryIds", value: ids.slice(0, 10)}], which leaves the other fields alone under concurrent reinforce/decay writes.That is still over 16 MiB, and
mem:semantic(31.8 MB) andmem:graph:edges(64.8 MB) can't shrink without deleting data. So this only helps; paging instate::list(or key-only listing plus batched gets) is still required. A one-time migration that capssourceMemoryIdson pre-0.9.31 insights would be a cheap companion fix.