Repository navigation
Description
mem::smart-search crashes the whole iii worker — one request unregisters all 271 functions for ~5s
Summary
A single request to POST /agentmemory/smart-search (or one call to the memory_smart_search MCP tool) takes down the entire iii worker, not just that function. Immediately after the request, every endpoint — including /agentmemory/health and /agentmemory/sessions — fails with Function ... not found. The worker reconnects on its own after roughly 5 seconds and everything works again. The node process itself does not restart (uptime keeps climbing).
The blast radius is what makes this worth reporting: the failing function is one endpoint, but the observable symptom is "agentmemory is broken" — the viewer hangs on load, session lists time out, and the dashboard shows its empty state ("agentmemory is running but hasn't seen any sessions yet") even though 339 sessions are stored and readable.
Environment
@agentmemory/agentmemory |
0.9.28 (global npm install) |
| iii engine | 0.11.2 (pinned) |
| Node | v26.5.0 |
| OS | Darwin 25.5.0, arm64 (macOS, Apple Silicon) |
| Store | file-based, ~/.agentmemory/data, ~503 MB |
| Scale | 339 sessions, 160 observation scopes, 300 lessons, 108 summaries, 3273 semantic memories, 404 crystals, 921 insights, graph 9474 nodes / 8663 edges |
| Embeddings | EMBEDDING_PROVIDER=openai → ollama bge-m3 @ localhost:11434/v1, 1024 dims |
| LLM | deepseek deepseek-v4-flash |
Reproduction
# 1. baseline — everything healthy
curl -s http://127.0.0.1:3111/agentmemory/health | jq '.health.workers[0].function_count' # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length' # 339
# 2. one smart-search request
curl -s -X POST http://127.0.0.1:3111/agentmemory/smart-search \
-H 'Content-Type: application/json' -d '{"query":"x","limit":1}'
# => HTTP 500 {"error":"Invocation stopped","error_id":"db326f1f8608"}
# 3. immediately after — the whole worker is gone
curl -s http://127.0.0.1:3111/agentmemory/health
# => HTTP 500 {"error":"Function middleware::api-auth not found"}
curl -s http://127.0.0.1:3111/agentmemory/sessions
# => HTTP 500 {"error":"Function api::sessions not found"}
# 4. wait ~5s — fully recovered, no restart needed
sleep 5
curl -s http://127.0.0.1:3111/agentmemory/health | jq '.health.workers[0].function_count' # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length' # 339Measured timeline:
| Moment | /agentmemory/health |
/agentmemory/sessions |
|---|---|---|
| before | 200, function_count: 271 |
200, 339 sessions, 0.0s |
right after one smart-search |
500 Function middleware::api-auth not found |
500 Function api::sessions not found |
| +5s | 200, function_count: 271 |
200, 339 sessions, 0.0s |
| +15s / +30s | 200 | 200 |
Reproduced on every attempt, with the crash occurring ~0.5–0.6s into the request.
Narrowing it down
Each case below was run only after waiting for function_count to be back at 271, so the results are independent:
| Request body | Result | Worker survived? |
|---|---|---|
{"query":"x","limit":1} |
500 Invocation stopped (0.6s) |
no |
{"query":"x","limit":1,"includeLessons":false} |
500 Invocation stopped (0.5s) |
no |
{"expandIds":["obs_..."]} |
200 {"mode":"expanded",...} |
yes |
{} |
400 query or expandIds is required |
yes |
So the expandIds branch and the validation path are both fine, and includeLessons: false does not help — which points at the searchFn(data.query, overFetchLimit) call in the query branch of mem::smart-search rather than at recallLessons.
Notably, plain POST /agentmemory/search with the same query returns 200 with results and leaves the worker alive, so the keyword/BM25 path is healthy. That isolates the fault to the hybrid/vector portion reached only via smart-search.
What is not the cause
Ruled out during diagnosis, in case it saves you time:
- Not the data.
memory_diagnosereports 0 failures (13 pass / 9 warn), all summaries/semantic/crystals/insights consistent, and/agentmemory/sessionsreturns all 339 sessions in 0.0s when the worker is up. - Not the store size. 339 sessions serialize instantly; there is no slow path here, only a crash.
- Not embeddings. ollama answers on
localhost:11434/v1,bge-m3:latestis present, andPOST /v1/embeddingsreturns a 1024-dim vector in 0.2s. - Not the LLM.
POST /chat/completionsagainst the configured deepseek endpoint returns 200 in 1.0s with a valid key. - Not a stale install. Fresh
npm install -g @agentmemory/agentmemory@latest, and the crash persists across a clean restart where the node process and the engine both start together.
Impact
Because the crash unregisters all functions, unrelated requests in flight or issued within that window fail in confusing ways:
- The viewer on
:3113hangs on load indefinitely. GET /agentmemory/sessionsissued inside the crash window never completes and eventually returns 504 after exactly 180s (default_timeout: 180000for theiii-httpworker iniii-config.yaml).- The dashboard renders its first-run empty state, telling the user to run the demo, while hundreds of real sessions sit in the store.
- The
memory_smart_searchMCP tool silently returns{"results": []}instead of surfacing the error, so agents get "no results" rather than "the backend just crashed" — and each call knocks the worker out for the next ~5 seconds.
Suggestions
- Contain the failure: an exception inside one registered function should not deregister the worker's whole function table.
- Surface it: have the MCP tool propagate the 500 instead of returning an empty result set.
- The root exception is presumably visible in the engine log against
error_id(e.g.db326f1f8608,db33065be0c0) — happy to attach that output, or run any instrumented build you point me at.
Activity
tagius commentedon Aug 20, 2026
Confirming this on a second machine, and adding a version boundary: on 0.9.29 the plain
POST /agentmemory/searchpath drops the worker too. The blast radius grew from the hybrid/vector path to the keyword path in one release.The OP isolates the fault to "the hybrid/vector portion reached only via
smart-search" because plain/searchstayed healthy on 0.9.28. That still holds on 0.9.28 here — but it stops holding on 0.9.29.Differential
Same store, same host. Engine and worker both fully restarted between arms (
launchctl kickstart -konly replaces the worker, and a worker left wedged by a previous failed search invalidates the next measurement — that cost me an hour, flagging it in case it saves someone else):POST /agentmemory/searchinvocation_stopped, ~2.7 sPOST /agentmemory/smart-searchinvocation_stopped, ~0.5 smem::insight-search(also vector-backed)mem::lesson-recallmem::auto-forget(walks all 46k observations)memory_saveReproduced on every attempt in both directions — I rolled 0.9.28 → 0.9.29 → 0.9.28 → 0.9.29 → 0.9.28 and the result tracked the version each time.
Environment
@agentmemory/agentmemory~/.agentmemory/data, 651 MBEMBEDDING_PROVIDER=local,Xenova/all-MiniLM-L6-v2, 384 dimsdeepseek/deepseek-v4-flashNote the scale relative to the OP's 339 sessions / ~503 MB — this store is meaningfully larger on the observation side, which I think is the relevant variable.
Why I think this is #1142, not a vector-path bug
#1142 reports
GET /agentmemory/exportdropping the worker once the response passes 16 MiB, 404ing every endpoint — same signature, with a size threshold attached. Everything here fits that better than it fits "the vector path is broken":mem::insight-searchis vector-backed and works fine on 0.9.29 — it just searches a much smaller set. So it isn't embeddings or the vector code path.mem::auto-forgetwalks all 46k observations on 0.9.29 without incident. So it isn't touching every observation either — it's how much data comes back.limitis no defence:mem::searchfails identically atlimit1, 3, and 10, always in ~0.5 s.limitcaps the returned results, not the intermediate payload the function hands the engine.Read that way, 0.9.29 didn't introduce a new bug so much as push
mem::search's response over a cap that 0.9.28 stays under at this corpus size — which would also explain why/searchis still fine for the OP on both versions at 339 sessions. If that's right, the OP's "not the store size" line is correct about serialization speed but the threshold is real, just on a different axis. It also means the set of affected endpoints grows with the store rather than being fixed per release.Things I ruled out on 0.9.29 before landing there, in case it saves anyone a repeat:
@xenova→@huggingface/transformersv4 migration (chore(deps): migrate @xenova/transformers to @huggingface/transformers v4 #1096). Loadingpipeline('feature-extraction','Xenova/all-MiniLM-L6-v2')straight out of the 0.9.29 tree gives 384 dims in 2.5 s cold.npm i -gwarns thatonnxruntime-node,sharpandprotobufjsinstall scripts didn't run; irrelevant, the native binaries ship prebuilt and both modulesrequire()fine.Two secondary symptoms worth folding into the impact list
1.
agentmemory statusreports zeros, which reads as data loss. After a run of failed searches the engine wedges, and status showsSessions: 0,Observations: 0,Health: critical— whileMemories: 78and the graph counts stay correct and/searchstill returns real observations. Nothing was lost:data/stayed byte-for-byte the size of an APFS clone taken before any of this, and a full engine + worker restart restored930 / 46,114exactly. But the first read of that screen is "my store is gone", which is a rough thing to show someone whose search just broke.2. Confirming the silent-empty MCP behaviour.
memory_smart_searchreturns{"results": []}rather than surfacing the 500, exactly as the OP describes. In my Claude Code MCP logs it shows asTool 'memory_smart_search' completed successfully in 7ms— a success line, 7 ms, over a 46k-observation store. There's no way for an agent to tell that apart from "genuinely no matches", so this had been silently failing here for an unknown length of time before I went looking for something else entirely. Seconding suggestion 2 in the OP strongly.Happy to run an instrumented build, or to dump
error_idvalues against engine logs — I have a store that reproduces the 0.9.29 case on demand, and the pre-upgrade clone to test against.Staying on 0.9.28 here for now, since
mem::searchis whatmemory_recalldepends on.fatesealed commentedon Sep 14, 2026
Additional reproduction on agentmemory 0.9.29 / iii-engine 0.11.2 / iii-sdk 0.11.2, Node v23.11.0, macOS 15.7.4 arm64 (global npm install, native engine, file-based KV, local embeddings).
After a complete engine + worker restart, Dashboard, Sessions, health, and MCP session listing were working. The store had 352 sessions and approximately 415 MiB of persisted data. Smart search still reproducibly failed:
Two attempts returned HTTP 500 after approximately 5.93 s and 5.21 s; the captured second response was:
{"error":"Invocation stopped","error_id":"289382fd8320"}The log then showed
[iii] Reconnecting ...and a new worker ID while the Node process PID stayed unchanged. The service recovered between attempts.Concrete size evidence for the engine -> SDK leg:
mem:graph:edgesfound 44,197 entries.new WebSocket(address, { headers })without amaxPayloadoverride. Its installedwsclient default is 104,857,600 bytes (100 MiB).HybridSearch.tripleStreamSearch -> GraphRetrieval.expandFromChunkscalls an unpaginatedkv.list(KV.graphEdges).searchByEntitiesalso enumerates the graph. The caller'slimit: 2does not bound these intermediate responses.The oversized intermediate graph response is therefore a strong explanation for the disconnect. We did not capture a WebSocket close code, so this is not a claim that close code 1009 was directly observed. This is the 100 MiB engine -> worker message limit, distinct from the 16 MiB worker -> engine frame limit discussed in #1142 and #1360.
Also confirmed in the installed code:
AGENTMEMORY_GRAPH_WEIGHT=0changes fusion scoring but does not skip graph retrieval, andGRAPH_EXTRACTION_ENABLED=falsedoes not disable this read path. The full-graph retrieval issue described in #1300 and the changes proposed in #1351 are relevant; any fix should also account for a first graph load already larger than the transport limit, not only repeated traversal cost.This was observed after recovering a separate long-running metrics-storage lockup reported in #1312. SDK metrics export was disabled during these search reproductions, so that workaround restores the viewer but does not fix the search-related disconnect.
Could graph retrieval use bounded/paged or per-node adjacency reads, and could an oversized result fail just the request without disconnecting the entire worker? A regression case above the 100 MiB intermediate-result limit would cover this reproduction.
sachin-fluid commentedon Sep 16, 2026
@rohitg00 This is a real issue and needs to be fixed. I'm running it pinned down to 0.9.28
Jonaxor commentedon Sep 17, 2026
Confirmed on a second environment — the bug is alive in 0.9.29 and the crash signature is identical.
Environment
@agentmemory/agentmemory0.9.29 (global npm install), iii engine (pinned via the package's bundled binary), Linux (Arch), Node via npmlocalhost:11434/v1(nomic-embed-text,qwen3:4b),OPENAI_TIMEOUT_MS=180000/api/tags200 in 32 ms,/v1/embeddingsreturns a full 1024-dim vector in <1 s, andPOST /agentmemory/search(BM25-only),/lessons/search,GET /sessionsall return 200Repro (2026-09-17)
Additional error_ids from earlier attempts today:
17d25ea28106,18a60266e165,17d410ae9522.Two observations that might help narrowing:
{"query":"…","limit":2,"types":["memory"]}) returnedFunction api::smart-search not foundinstead ofInvocation stopped— consistent with the worker being unregistered rather than the function itself failing a second time.OPENAI_*env of the worker process.This rules out macOS-specific worker behavior, the
bge-m3/deepseekprovider combination, and large store size as necessary conditions — on this box the store is modest and models are different, butmem::smart-searchstill takes the whole worker down the same way. Hoping the fix-shape suggestion in the issue body (contain the failed function instead of deregistering the worker's table) gets picked up; it would make this survivable even if the vector leg keeps throwing.fix(state): stop oversized payloads from dropping the worker (rohitg0…
JCokP commentedon Sep 25, 2026
Confirming @fatesealed's graph-size explanation: I fixed it by removing the graph.
Env: 0.9.29, iii 0.11.2, Node v22.23.2, macOS 26.6 arm64, local embeddings, deepseek-v4-flash.
Graph size:
mem:graph:nodes194 MB (25,260 entries),mem:graph:edges320 MB (54,788 entries). That is well over the 100 MiBwslimit. Most of the size issourceObservationIds, which never shrinks: median ~200 ids per node, max 5,175./graph/statsshowed only 563 nodes because it reads the snapshot.Fix: I stopped the service, moved the
mem%3Agraph%3A*.binfiles out ofstate_store.db/, and restarted.POST /smart-searchInvocation stopped, worker dropsPOST /reflectInvocation stopped, worker drops/graph/resetdoesn't help here. It only writes an empty snapshot and leaves the node/edge rows in place.Also affected:
mem::reflectalso runskv.liston the whole graph. The scheduled reflect had timed out on every run (145 in a row).The graph grows back:
GRAPH_EXTRACTION_ENABLED=falseonly gates the LLM pass.event::session::stoppedstill firesmem::graph-extractunconditionally; one session end after the cleanup added 896 nodes. My local patch:Suggestions: make graph reads in
GraphRetrievalandmem::reflectbounded (see #1300 / #1351), capsourceObservationIds, and make the flag also disable structural extraction.feat(graph): compact bloated graph provenance once at boot (#1455)
rohitg00 commentedon Oct 8, 2026
@i-sannikov fixed in v0.9.30: bounded graph provenance stops huge rows from crash-looping the worker during search, and
AGENTMEMORY_GRAPH_WEIGHT=0now skips graph retrieval entirely. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30