You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Oversized store fields and unpaginated full-list endpoints: sourceSessionIds/sourceMemoryIds bloat + 16 MiB WS frame limit #1360
Hi! We run @agentmemory/agentmemory v0.9.27 + iii v0.11.2 (self-hosted, file_based KV, Node 20) in production for ~3 months. Three related problems, all caused by unbounded data growth:
1. mem::consolidate copies all recentSummaries into every new fact's sourceSessionIds
(index.mjs:8486: sourceSessionIds: recentSummaries.map(s => s.sessionId)). In our store this field reached 56% of mem:semantic (8.96 MB): 344,577 references to only 41 unique sessionIds (8404× duplication) — the same ~20 old sessions, because recentSummaries barely changes. The field is only read via .includes() (:13501), so the duplication is pure waste. Suggestion: cap to the last N, or don't duplicate already-known ids.
2. mem::reflect copies cluster.factIds wholesale into each insight's sourceMemoryIds
(index.mjs:13099). Ours grew to 418 MB in 3 months (~14.2M refs; median 501, max 3952 per insight), while sourceMemoryIds is never read back. At ~418 MB the worker connection dies (cf. #1153, #1124). Suggestion:slice(0, N), or store only sourceConceptCluster.
3. Full-list endpoints have no pagination, and the 16 MiB response frame limit kills the worker. api::semantic-list (:17140), /memories, /export (:16481), /mesh/export without since return the whole list in one response body. When that exceeds max_frame_size (16 MiB, axum/tungstenite) the function result can't be sent: we get HTTP 500 Invocation stopped, the worker drops, all 271 functions 404 for ~1s, then [iii] Reconnecting… Worker registered. Repro: GET /agentmemory/insights?limit=30000 (≈23 MB body) → 500.
Note there are two distinct WS limits: engine→worker kv.list result (Node wsmaxPayload, 100 MiB — this is what #1124 hits at large sizes) vs worker→engine function result (16 MiB). Both need protection, but by different means.
Suggestion: mandatory limit/since (or a sane default cap) on all full-list endpoints, plus bounds on the two bloat fields above.
Update: same pattern found in mem:graph:edges (sourceObservationIds)
Following up with a fourth instance of the same "unbounded field growth via merge" pattern,
this time in the knowledge graph.
Where:mem::graph-extract (index.mjs:8102-8180) is triggered per-session
(event::session::stopped → isGraphExtractionEnabled()), with no cap on the number of
nodes/edges an LLM extraction call can return — unlike mem::reflect, which caps at maxClusters: 10 (max 20, :12944-13040).
Edge deduplication IS exact-match on sourceNodeId|targetNodeId|type
(edgeIndexKey, :7947) — we verified 0 duplicate edges by that key. But when the same
entity pair recurs in a later session, mergeEdge() fires:
sourceObservationIds grows without any upper bound on every re-extraction that touches an
already-known pair, and createdAt is never refreshed on merge — so old edges silently bloat
over time rather than new edges being created. In our store (47,711→48,047 edges during this
session): sourceObservationIds = 50.3% of the file's JSON size (5.68 MB of ~17 MB). Five
"hot" observation ids (created 2026-08-30..09-03, likely a frequently-referenced README/core
file) accounted for 26,445 of 195,823 total references (up to 435× duplication on some
edges) — a handful of commonly-cited sessions keep re-accumulating links on every subsequent
extraction that mentions the same entities.
No retention/GC applies to the graph.mem::retention-score/mem::retention-evict
(:14109-14268) only operate on KV.retentionScores by memoryId (plain memories). The only
code that ever sets edge.stale = true is mem::cascade-update (:12341-12379), and only as
a side effect of superseding a plain memory. In our corpus stale was 0 across all 47k+ edges —
cascade-update had simply never fired for the affected observations.
Field is write-only in the read paths that matter. We confirmed sourceObservationIds is
never read by mem::graph-query, paginate/paginateFromSnapshot, or snapshot rebuild — only
by cascade-update's matching logic. This means capping it is safe for graph-query/smart-search/
reflect; the only cost is reduced cascade-update recall (fewer graph elements get marked stale
when an old memory is superseded).
Mitigation applied on our side (self-hosted, stopgap only): capped sourceObservationIds to
the last 3 elements per edge via decode→edit→encode of the .bin store (not a code patch).
47,711→48,047 edges, 23,831 had >3 refs, removed 72,859 of 198,247 total references. mem:graph:edges.bin 17.11 MiB → ~14.4-15.1 MiB. This buys time but does not stop the
underlying growth rate (~25%/night observed before cleanup) — it needs to be repeated
periodically until there's a code fix.
Suggested fixes (same family as the ones already proposed above):
maxEdgesPerExtract / maxNewEdgesPerSession cap in mem::graph-extract, mirroring reflect's maxClusters.
Bound/ring-buffer sourceObservationIds directly in mergeEdge/mergeNode (e.g. keep last N),
rather than relying on downstream store surgery.
Extend retention-evict to graphNodes/graphEdges, e.g. by staleness + low node-degree.
api::mesh-export without since (:18052-18094) currently does df(graphEdges, "createdAt")
with no pagination or size guard — it should reject with a clear 400/413 instead of silently
returning a body that exceeds the 16 MiB function-response frame limit and crashing the worker
connection (as already described in point 3 above).
Happy to share the decode/cap script or more store samples if useful.
Independent confirmation of point 2 on v0.9.29 + iii v0.11.2 (Windows 11, file_based KV), from a much smaller store — which makes the ratio easier to see.
mem:insights was 32.5 MB for 1,921 insights on a store holding 46 memories. Sampling 50 insights:
field
average
max
content
381 chars
494
sourceMemoryIds
1,072 ids
1,513
sourceLessonIds
14
32
sourceConceptCluster
94
128
So ~97% of each record is provenance that, as you say, is never read back: grepping both bundles, sourceMemoryIds appears only at write sites (sourceMemoryIds: cluster.factIds in reflect, data.sourceMemoryIds || [] on the create path) — no .sourceMemoryIds reader anywhere.
We applied your suggested fix:
sourceMemoryIds: cluster.factIds.slice(-50),
New insights are now ~50 ids; the old ones we could only drop wholesale, since there is no per-record update endpoint.
Same root cause as #1171 on the graph side, and we hit it there too — 3.1M obs_* references across 5,004 nodes, 145 MB of mem:graph:*, with Invocation timeout after 180000ms: state::set on every extraction once the collection file got large enough. Both are the same shape of bug: a provenance array unioned without a bound, in a store that rewrites the whole collection per write. A shared cap constant across the three sites (semantic sourceSessionIds, insight sourceMemoryIds, graph sourceObservationIds) would close the whole class.
Confirming point 3 — and a note on what 0.9.30's new guard does and does not fix.
On v0.9.30 (Windows 11, file_based KV) /agentmemory/export no longer 500s with Invocation stopped: it is bounded by SAFE_PAYLOAD_BYTES = 15 * 1024 * 1024 (index.mjs:5786) and degrades to a warning plus an oversized-payload error instead:
[agentmemory] warn Export exceeds transport frame limit {"bytesSoFar":15729039,"collection":"summaries"}
That is a real improvement — the worker survives, which it did not on 0.9.27. But it is not a workaround, because the collection it trips on cannot be narrowed:
mem::export reads summaries with a plain kv.list(KV.summaries) and refuses if it does not fit (index.mjs:12386-12387). No limit, no since, no cursor.
?maxSessions / ?offset only slice sessions (:12382).
EXPORT_OVERSIZE_HINT (:12330) says "or export fewer collections", but the handler reads no collection filter — that parameter does not exist.
So once the summaries collection alone crosses 15 MiB, /export is dead with no way out. Ours is 15,729,039 B — 399 B over the limit when it first tripped, so the margin is not theoretical. summaries also is not in the hint's list of unpaginated collections, which made it non-obvious which one blew the budget.
/semantic and /procedural are tracked in #1447; this looks like a fourth instance of the same family. Happy to test a patch — a limit/since on summaries, or making the hint true by adding a real collections= filter — if that is useful.
#1142 (the Invocation stopped / 500 form of this) was closed on 2026-10-08, with the 0.9.30 frame guard as the rationale. The guard does stop the daemon from dying, and that is real progress. But it converts a crash into a permanent refusal, and that refusal is not escapable on the released version:
GET /agentmemory/export -> HTTP 413
{"bytes":15729297,"limitBytes":15728640,"stoppedAt":"summaries","success":false}
657 bytes over, tripping on summaries. ?maxSessions / ?offset page sessions only; summaries, memories and the sixteen top-level collections are read whole, and mem::export refuses the entire payload rather than trimming it. So this store cannot be exported at any parameter -- the backup/migration path is closed, not narrowed.
The hint text (or export fewer collections) describes a parameter that does not exist on v0.9.30. That is what #1143 adds: ?collectionLimit / ?collectionOffset / ?collections. It is open, MERGEABLE, and every test job is green (the only red check is the Vercel deploy, for an unrelated team-authorization reason). It has been sitting since 2026-08-03.
Nothing to add beyond that -- just flagging that #1360's export point is still open on the released version, and that the fix already exists.
Data point for point 2 on 0.9.30. PR #1295 covers about half of it; the other half comes from two concept arrays that nobody has reported yet.
mem:insights.bin is 191 MB with 4,763 insights, and it is growing faster each day: 27.7 MB of new insights today from 77 insights, about 360 KB each. Bytes per field:
an exact copy of sourceConceptCluster in all 4,763 records (tags: conceptNames and sourceConceptCluster: conceptNames in mem::reflect)
What reads these fields in the 0.9.30 dist:
sourceConceptCluster is never read.
tags is read only by mem::insight-search, which joins it into a keyword haystack.
Dedup and reinforcement are keyed on the content fingerprint, not on the cluster, so capping the arrays doesn't change which insights get created.
The cluster sizes look like the mega-clusters from #1133.
Suggestion, beyond #1295: cap conceptNames (or just tags) at a few dozen entries when mem::reflect writes an insight, and drop the duplicate sourceConceptCluster or store it once. Capping the id arrays to their last 20 and the concept arrays to their first 50 shrinks our existing file from 191 MB to 19 MB.
Past ~100 MiB, every full list of insights drops the worker: /insights, insight-search, mem::diagnose, and the 24 h insight-decay-sweep. So a store this size is a time bomb even when nobody calls those endpoints.
Confirming this still happens on 0.9.33 + iii 0.25.1, with two paging details that cost me a while to work out.
/export still 500s on a modest store. 2,718 sessions / 8,384 observations / 1,824 memories:
GET /agentmemory/export
500 {"error":{"code":"payload_too_large",
"message":"Serialized JSON envelope is 23525385 bytes; target worker accepts up to 16777216 bytes."}}
Paging does work, but the parameters are not symmetric:
collections=sessions does not page sessions. Asking for it alone returns all 2,718 sessions (~12.5 MiB) regardless of collectionLimit. It only pages when combined with maxSessions/offset. So ?collections=sessions&collectionLimit=10 silently ignores the limit.
collectionPagination.totals is a reliable reconciliation target for every collection except accessLogs (see #1792).
Also worth noting for the "two distinct WS limits" point in the OP: state.list through the CLI on a 16.8 MB scope succeeds, while the same scope's content through a worker function result does not. So the two limits really do behave differently, and the 16 MiB one is the practical ceiling to design against.
Activity
it-space13 commentedon Sep 11, 2026
Update: same pattern found in
mem:graph:edges(sourceObservationIds)Following up with a fourth instance of the same "unbounded field growth via merge" pattern,
this time in the knowledge graph.
Where:
mem::graph-extract(index.mjs:8102-8180) is triggered per-session(
event::session::stopped→isGraphExtractionEnabled()), with no cap on the number ofnodes/edges an LLM extraction call can return — unlike
mem::reflect, which caps atmaxClusters: 10(max 20,:12944-13040).Edge deduplication IS exact-match on
sourceNodeId|targetNodeId|type(
edgeIndexKey,:7947) — we verified 0 duplicate edges by that key. But when the sameentity pair recurs in a later session,
mergeEdge()fires:sourceObservationIdsgrows without any upper bound on every re-extraction that touches analready-known pair, and
createdAtis never refreshed on merge — so old edges silently bloatover time rather than new edges being created. In our store (47,711→48,047 edges during this
session):
sourceObservationIds= 50.3% of the file's JSON size (5.68 MB of ~17 MB). Five"hot" observation ids (created 2026-08-30..09-03, likely a frequently-referenced README/core
file) accounted for 26,445 of 195,823 total references (up to 435× duplication on some
edges) — a handful of commonly-cited sessions keep re-accumulating links on every subsequent
extraction that mentions the same entities.
No retention/GC applies to the graph.
mem::retention-score/mem::retention-evict(
:14109-14268) only operate onKV.retentionScoresbymemoryId(plain memories). The onlycode that ever sets
edge.stale = trueismem::cascade-update(:12341-12379), and only asa side effect of superseding a plain memory. In our corpus
stalewas 0 across all 47k+ edges —cascade-update had simply never fired for the affected observations.
Field is write-only in the read paths that matter. We confirmed
sourceObservationIdsisnever read by
mem::graph-query,paginate/paginateFromSnapshot, or snapshot rebuild — onlyby
cascade-update's matching logic. This means capping it is safe for graph-query/smart-search/reflect; the only cost is reduced
cascade-updaterecall (fewer graph elements get marked stalewhen an old memory is superseded).
Mitigation applied on our side (self-hosted, stopgap only): capped
sourceObservationIdstothe last 3 elements per edge via decode→edit→encode of the
.binstore (not a code patch).47,711→48,047 edges, 23,831 had >3 refs, removed 72,859 of 198,247 total references.
mem:graph:edges.bin17.11 MiB → ~14.4-15.1 MiB. This buys time but does not stop theunderlying growth rate (~25%/night observed before cleanup) — it needs to be repeated
periodically until there's a code fix.
Suggested fixes (same family as the ones already proposed above):
maxEdgesPerExtract/maxNewEdgesPerSessioncap inmem::graph-extract, mirroringreflect'smaxClusters.sourceObservationIdsdirectly inmergeEdge/mergeNode(e.g. keep last N),rather than relying on downstream store surgery.
retention-evicttographNodes/graphEdges, e.g. by staleness + low node-degree.api::mesh-exportwithoutsince(:18052-18094) currently doesdf(graphEdges, "createdAt")with no pagination or size guard — it should reject with a clear 400/413 instead of silently
returning a body that exceeds the 16 MiB function-response frame limit and crashing the worker
connection (as already described in point 3 above).
Happy to share the decode/cap script or more store samples if useful.
docs(triage): keep five iii-symptom rows as candidates
docs(triage): keep five iii-symptom rows as candidates
sijuz commentedon Oct 5, 2026
Independent confirmation of point 2 on v0.9.29 + iii v0.11.2 (Windows 11,
file_basedKV), from a much smaller store — which makes the ratio easier to see.mem:insightswas 32.5 MB for 1,921 insights on a store holding 46 memories. Sampling 50 insights:contentsourceMemoryIdssourceLessonIdssourceConceptClusterSo ~97% of each record is provenance that, as you say, is never read back: grepping both bundles,
sourceMemoryIdsappears only at write sites (sourceMemoryIds: cluster.factIdsin reflect,data.sourceMemoryIds || []on the create path) — no.sourceMemoryIdsreader anywhere.We applied your suggested fix:
New insights are now ~50 ids; the old ones we could only drop wholesale, since there is no per-record update endpoint.
Same root cause as #1171 on the graph side, and we hit it there too — 3.1M
obs_*references across 5,004 nodes, 145 MB ofmem:graph:*, withInvocation timeout after 180000ms: state::seton every extraction once the collection file got large enough. Both are the same shape of bug: a provenance array unioned without a bound, in a store that rewrites the whole collection per write. A shared cap constant across the three sites (semanticsourceSessionIds, insightsourceMemoryIds, graphsourceObservationIds) would close the whole class.rohitg00 commentedon Oct 8, 2026
@it-space13 partially addressed in v0.9.30: source ids are capped,
GET /agentmemory/memoriesis paginated withnextCursor, and export refuses oversized payloads./semanticand/proceduralare still unpaginated; #1447 covers them. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.307889545 commentedon Oct 12, 2026
Confirming point 3 — and a note on what 0.9.30's new guard does and does not fix.
On v0.9.30 (Windows 11,
file_basedKV)/agentmemory/exportno longer 500s withInvocation stopped: it is bounded bySAFE_PAYLOAD_BYTES = 15 * 1024 * 1024(index.mjs:5786) and degrades to a warning plus an oversized-payload error instead:That is a real improvement — the worker survives, which it did not on 0.9.27. But it is not a workaround, because the collection it trips on cannot be narrowed:
mem::exportreadssummarieswith a plainkv.list(KV.summaries)and refuses if it does not fit (index.mjs:12386-12387). Nolimit, nosince, no cursor.?maxSessions/?offsetonly slicesessions(:12382).EXPORT_OVERSIZE_HINT(:12330) says "or export fewer collections", but the handler reads no collection filter — that parameter does not exist.So once the summaries collection alone crosses 15 MiB,
/exportis dead with no way out. Ours is 15,729,039 B — 399 B over the limit when it first tripped, so the margin is not theoretical.summariesalso is not in the hint's list of unpaginated collections, which made it non-obvious which one blew the budget./semanticand/proceduralare tracked in #1447; this looks like a fourth instance of the same family. Happy to test a patch — alimit/sinceonsummaries, or making the hint true by adding a realcollections=filter — if that is useful.7889545 commentedon Oct 12, 2026
Follow-up on the export point, one release later.
#1142 (the
Invocation stopped/ 500 form of this) was closed on 2026-10-08, with the 0.9.30 frame guard as the rationale. The guard does stop the daemon from dying, and that is real progress. But it converts a crash into a permanent refusal, and that refusal is not escapable on the released version:657 bytes over, tripping on
summaries.?maxSessions/?offsetpagesessionsonly;summaries,memoriesand the sixteen top-level collections are read whole, andmem::exportrefuses the entire payload rather than trimming it. So this store cannot be exported at any parameter -- the backup/migration path is closed, not narrowed.The hint text (
or export fewer collections) describes a parameter that does not exist on v0.9.30. That is what #1143 adds:?collectionLimit/?collectionOffset/?collections. It is open, MERGEABLE, and every test job is green (the only red check is the Vercel deploy, for an unrelated team-authorization reason). It has been sitting since 2026-08-03.Nothing to add beyond that -- just flagging that #1360's export point is still open on the released version, and that the fix already exists.
aberenyi commentedon Oct 12, 2026
Data point for point 2 on 0.9.30. PR #1295 covers about half of it; the other half comes from two concept arrays that nobody has reported yet.
mem:insights.binis 191 MB with 4,763 insights, and it is growing faster each day: 27.7 MB of new insights today from 77 insights, about 360 KB each. Bytes per field:sourceMemoryIdssourceConceptClustertagssourceConceptClusterin all 4,763 records (tags: conceptNamesandsourceConceptCluster: conceptNamesinmem::reflect)What reads these fields in the 0.9.30 dist:
sourceConceptClusteris never read.tagsis read only bymem::insight-search, which joins it into a keyword haystack.The cluster sizes look like the mega-clusters from #1133.
Suggestion, beyond #1295: cap
conceptNames(or justtags) at a few dozen entries whenmem::reflectwrites an insight, and drop the duplicatesourceConceptClusteror store it once. Capping the id arrays to their last 20 and the concept arrays to their first 50 shrinks our existing file from 191 MB to 19 MB.Past ~100 MiB, every full list of insights drops the worker:
/insights,insight-search,mem::diagnose, and the 24 hinsight-decay-sweep. So a store this size is a time bomb even when nobody calls those endpoints.hptitan commentedon Nov 15, 2026
Confirming this still happens on 0.9.33 + iii 0.25.1, with two paging details that cost me a while to work out.
/exportstill 500s on a modest store. 2,718 sessions / 8,384 observations / 1,824 memories:Paging does work, but the parameters are not symmetric:
collections=sessionsdoes not page sessions. Asking for it alone returns all 2,718 sessions (~12.5 MiB) regardless ofcollectionLimit. It only pages when combined withmaxSessions/offset. So?collections=sessions&collectionLimit=10silently ignores the limit.collections=accessLogsalone returns[]. It is derived from the memories in the same response (exportedMemoryIds,export-import.ts:216-233), so it has to be requested together withmemories. And because that filter only knowsmem_*ids, observation-level and lesson-level access logs never come out at all — separate issue filed as audit storage is 92% repeated prompt text (98.5% of the 16 MiB frame limit) and export drops 84% of access logs #1792.What worked, for anyone else hitting this:
collectionPagination.totalsis a reliable reconciliation target for every collection exceptaccessLogs(see #1792).Also worth noting for the "two distinct WS limits" point in the OP:
state.listthrough the CLI on a 16.8 MB scope succeeds, while the same scope's content through a worker function result does not. So the two limits really do behave differently, and the 16 MiB one is the practical ceiling to design against.