Skip to content

mem::smart-search crashes the whole iii worker — one request unregisters all 271 functions for ~5s #1124

Closed

Description

@i-sannikov

mem::smart-search crashes the whole iii worker — one request unregisters all 271 functions for ~5s

Summary

A single request to POST /agentmemory/smart-search (or one call to the memory_smart_search MCP tool) takes down the entire iii worker, not just that function. Immediately after the request, every endpoint — including /agentmemory/health and /agentmemory/sessions — fails with Function ... not found. The worker reconnects on its own after roughly 5 seconds and everything works again. The node process itself does not restart (uptime keeps climbing).

The blast radius is what makes this worth reporting: the failing function is one endpoint, but the observable symptom is "agentmemory is broken" — the viewer hangs on load, session lists time out, and the dashboard shows its empty state ("agentmemory is running but hasn't seen any sessions yet") even though 339 sessions are stored and readable.

Environment

@agentmemory/agentmemory 0.9.28 (global npm install)
iii engine 0.11.2 (pinned)
Node v26.5.0
OS Darwin 25.5.0, arm64 (macOS, Apple Silicon)
Store file-based, ~/.agentmemory/data, ~503 MB
Scale 339 sessions, 160 observation scopes, 300 lessons, 108 summaries, 3273 semantic memories, 404 crystals, 921 insights, graph 9474 nodes / 8663 edges
Embeddings EMBEDDING_PROVIDER=openai → ollama bge-m3 @ localhost:11434/v1, 1024 dims
LLM deepseek deepseek-v4-flash

Reproduction

# 1. baseline — everything healthy
curl -s http://127.0.0.1:3111/agentmemory/health   | jq '.health.workers[0].function_count'   # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length'                  # 339

# 2. one smart-search request
curl -s -X POST http://127.0.0.1:3111/agentmemory/smart-search \
  -H 'Content-Type: application/json' -d '{"query":"x","limit":1}'
# => HTTP 500 {"error":"Invocation stopped","error_id":"db326f1f8608"}

# 3. immediately after — the whole worker is gone
curl -s http://127.0.0.1:3111/agentmemory/health
# => HTTP 500 {"error":"Function middleware::api-auth not found"}
curl -s http://127.0.0.1:3111/agentmemory/sessions
# => HTTP 500 {"error":"Function api::sessions not found"}

# 4. wait ~5s — fully recovered, no restart needed
sleep 5
curl -s http://127.0.0.1:3111/agentmemory/health   | jq '.health.workers[0].function_count'   # 271
curl -s http://127.0.0.1:3111/agentmemory/sessions | jq '.sessions | length'                  # 339

Measured timeline:

Moment /agentmemory/health /agentmemory/sessions
before 200, function_count: 271 200, 339 sessions, 0.0s
right after one smart-search 500 Function middleware::api-auth not found 500 Function api::sessions not found
+5s 200, function_count: 271 200, 339 sessions, 0.0s
+15s / +30s 200 200

Reproduced on every attempt, with the crash occurring ~0.5–0.6s into the request.

Narrowing it down

Each case below was run only after waiting for function_count to be back at 271, so the results are independent:

Request body Result Worker survived?
{"query":"x","limit":1} 500 Invocation stopped (0.6s) no
{"query":"x","limit":1,"includeLessons":false} 500 Invocation stopped (0.5s) no
{"expandIds":["obs_..."]} 200 {"mode":"expanded",...} yes
{} 400 query or expandIds is required yes

So the expandIds branch and the validation path are both fine, and includeLessons: false does not help — which points at the searchFn(data.query, overFetchLimit) call in the query branch of mem::smart-search rather than at recallLessons.

Notably, plain POST /agentmemory/search with the same query returns 200 with results and leaves the worker alive, so the keyword/BM25 path is healthy. That isolates the fault to the hybrid/vector portion reached only via smart-search.

What is not the cause

Ruled out during diagnosis, in case it saves you time:

  • Not the data. memory_diagnose reports 0 failures (13 pass / 9 warn), all summaries/semantic/crystals/insights consistent, and /agentmemory/sessions returns all 339 sessions in 0.0s when the worker is up.
  • Not the store size. 339 sessions serialize instantly; there is no slow path here, only a crash.
  • Not embeddings. ollama answers on localhost:11434/v1, bge-m3:latest is present, and POST /v1/embeddings returns a 1024-dim vector in 0.2s.
  • Not the LLM. POST /chat/completions against the configured deepseek endpoint returns 200 in 1.0s with a valid key.
  • Not a stale install. Fresh npm install -g @agentmemory/agentmemory@latest, and the crash persists across a clean restart where the node process and the engine both start together.

Impact

Because the crash unregisters all functions, unrelated requests in flight or issued within that window fail in confusing ways:

  • The viewer on :3113 hangs on load indefinitely.
  • GET /agentmemory/sessions issued inside the crash window never completes and eventually returns 504 after exactly 180s (default_timeout: 180000 for the iii-http worker in iii-config.yaml).
  • The dashboard renders its first-run empty state, telling the user to run the demo, while hundreds of real sessions sit in the store.
  • The memory_smart_search MCP tool silently returns {"results": []} instead of surfacing the error, so agents get "no results" rather than "the backend just crashed" — and each call knocks the worker out for the next ~5 seconds.

Suggestions

  1. Contain the failure: an exception inside one registered function should not deregister the worker's whole function table.
  2. Surface it: have the MCP tool propagate the 500 instead of returning an empty result set.
  3. The root exception is presumably visible in the engine log against error_id (e.g. db326f1f8608, db33065be0c0) — happy to attach that output, or run any instrumented build you point me at.

Activity

  1. tagius commented on Aug 20, 2026

    @tagius

    Confirming this on a second machine, and adding a version boundary: on 0.9.29 the plain POST /agentmemory/search path drops the worker too. The blast radius grew from the hybrid/vector path to the keyword path in one release.

    The OP isolates the fault to "the hybrid/vector portion reached only via smart-search" because plain /search stayed healthy on 0.9.28. That still holds on 0.9.28 here — but it stops holding on 0.9.29.

    Differential

    Same store, same host. Engine and worker both fully restarted between arms (launchctl kickstart -k only replaces the worker, and a worker left wedged by a previous failed search invalidates the next measurement — that cost me an hour, flagging it in case it saves someone else):

    0.9.28 0.9.29
    POST /agentmemory/search 200, real results, ~20 ms 500 invocation_stopped, ~2.7 s
    POST /agentmemory/smart-search 500 invocation_stopped, ~0.5 s 500 (then 404 — worker already gone)
    mem::insight-search (also vector-backed) ok ok
    mem::lesson-recall ok ok
    mem::auto-forget (walks all 46k observations) ok ok
    capture / compress / memory_save ok ok

    Reproduced on every attempt in both directions — I rolled 0.9.28 → 0.9.29 → 0.9.28 → 0.9.29 → 0.9.28 and the result tracked the version each time.

    Environment

    @agentmemory/agentmemory 0.9.28 and 0.9.29, global npm
    iii engine 0.11.2
    Node v25.4.0 (fnm)
    OS Darwin 25.6.0, arm64
    Store ~/.agentmemory/data, 651 MB
    Scale 930 sessions, 46,121 observations, 78 memories, graph 16,835 nodes / 21,087 edges
    Embeddings EMBEDDING_PROVIDER=local, Xenova/all-MiniLM-L6-v2, 384 dims
    LLM openrouter deepseek/deepseek-v4-flash

    Note the scale relative to the OP's 339 sessions / ~503 MB — this store is meaningfully larger on the observation side, which I think is the relevant variable.

    Why I think this is #1142, not a vector-path bug

    #1142 reports GET /agentmemory/export dropping the worker once the response passes 16 MiB, 404ing every endpoint — same signature, with a size threshold attached. Everything here fits that better than it fits "the vector path is broken":

    • mem::insight-search is vector-backed and works fine on 0.9.29 — it just searches a much smaller set. So it isn't embeddings or the vector code path.
    • mem::auto-forget walks all 46k observations on 0.9.29 without incident. So it isn't touching every observation either — it's how much data comes back.
    • limit is no defence: mem::search fails identically at limit 1, 3, and 10, always in ~0.5 s. limit caps the returned results, not the intermediate payload the function hands the engine.

    Read that way, 0.9.29 didn't introduce a new bug so much as push mem::search's response over a cap that 0.9.28 stays under at this corpus size — which would also explain why /search is still fine for the OP on both versions at 339 sessions. If that's right, the OP's "not the store size" line is correct about serialization speed but the threshold is real, just on a different axis. It also means the set of affected endpoints grows with the store rather than being fixed per release.

    Things I ruled out on 0.9.29 before landing there, in case it saves anyone a repeat:

    • Not the @xenova → @huggingface/transformers v4 migration (chore(deps): migrate @xenova/transformers to @huggingface/transformers v4 #1096). Loading pipeline('feature-extraction','Xenova/all-MiniLM-L6-v2') straight out of the 0.9.29 tree gives 384 dims in 2.5 s cold.
    • Not the blocked postinstalls. npm i -g warns that onnxruntime-node, sharp and protobufjs install scripts didn't run; irrelevant, the native binaries ship prebuilt and both modules require() fine.

    Two secondary symptoms worth folding into the impact list

    1. agentmemory status reports zeros, which reads as data loss. After a run of failed searches the engine wedges, and status shows Sessions: 0, Observations: 0, Health: critical — while Memories: 78 and the graph counts stay correct and /search still returns real observations. Nothing was lost: data/ stayed byte-for-byte the size of an APFS clone taken before any of this, and a full engine + worker restart restored 930 / 46,114 exactly. But the first read of that screen is "my store is gone", which is a rough thing to show someone whose search just broke.

    2. Confirming the silent-empty MCP behaviour. memory_smart_search returns {"results": []} rather than surfacing the 500, exactly as the OP describes. In my Claude Code MCP logs it shows as Tool 'memory_smart_search' completed successfully in 7ms — a success line, 7 ms, over a 46k-observation store. There's no way for an agent to tell that apart from "genuinely no matches", so this had been silently failing here for an unknown length of time before I went looking for something else entirely. Seconding suggestion 2 in the OP strongly.

    Happy to run an instrumented build, or to dump error_id values against engine logs — I have a store that reproduces the 0.9.29 case on demand, and the pre-upgrade clone to test against.

    Staying on 0.9.28 here for now, since mem::search is what memory_recall depends on.

  2. fatesealed commented on Sep 14, 2026

    @fatesealed

    Additional reproduction on agentmemory 0.9.29 / iii-engine 0.11.2 / iii-sdk 0.11.2, Node v23.11.0, macOS 15.7.4 arm64 (global npm install, native engine, file-based KV, local embeddings).

    After a complete engine + worker restart, Dashboard, Sessions, health, and MCP session listing were working. The store had 352 sessions and approximately 415 MiB of persisted data. Smart search still reproducibly failed:

    POST /agentmemory/smart-search
    Content-Type: application/json
    
    {"query":"agentmemory","limit":2}

    Two attempts returned HTTP 500 after approximately 5.93 s and 5.21 s; the captured second response was:

    {"error":"Invocation stopped","error_id":"289382fd8320"}

    The log then showed [iii] Reconnecting ... and a new worker ID while the Node process PID stayed unchanged. The service recovered between attempts.

    Concrete size evidence for the engine -> SDK leg:

    • A read-only inspection of mem:graph:edges found 44,197 entries.
    • Locally serializing the stored values as compact UTF-8 JSON produced 163,333,595 bytes (~155.77 MiB), excluding the invocation-response envelope. The actual wire response was not captured.
    • The installed SDK constructs new WebSocket(address, { headers }) without a maxPayload override. Its installed ws client default is 104,857,600 bytes (100 MiB).
    • HybridSearch.tripleStreamSearch -> GraphRetrieval.expandFromChunks calls an unpaginated kv.list(KV.graphEdges). searchByEntities also enumerates the graph. The caller's limit: 2 does not bound these intermediate responses.

    The oversized intermediate graph response is therefore a strong explanation for the disconnect. We did not capture a WebSocket close code, so this is not a claim that close code 1009 was directly observed. This is the 100 MiB engine -> worker message limit, distinct from the 16 MiB worker -> engine frame limit discussed in #1142 and #1360.

    Also confirmed in the installed code: AGENTMEMORY_GRAPH_WEIGHT=0 changes fusion scoring but does not skip graph retrieval, and GRAPH_EXTRACTION_ENABLED=false does not disable this read path. The full-graph retrieval issue described in #1300 and the changes proposed in #1351 are relevant; any fix should also account for a first graph load already larger than the transport limit, not only repeated traversal cost.

    This was observed after recovering a separate long-running metrics-storage lockup reported in #1312. SDK metrics export was disabled during these search reproductions, so that workaround restores the viewer but does not fix the search-related disconnect.

    Could graph retrieval use bounded/paged or per-node adjacency reads, and could an oversized result fail just the request without disconnecting the entire worker? A regression case above the 100 MiB intermediate-result limit would cover this reproduction.

  3. sachin-fluid commented on Sep 16, 2026

    @sachin-fluid

    @rohitg00 This is a real issue and needs to be fixed. I'm running it pinned down to 0.9.28

  4. Jonaxor commented on Sep 17, 2026

    @Jonaxor

    Confirmed on a second environment — the bug is alive in 0.9.29 and the crash signature is identical.

    Environment

    • @agentmemory/agentmemory 0.9.29 (global npm install), iii engine (pinned via the package's bundled binary), Linux (Arch), Node via npm
    • Embeddings + LLM: local Ollama at localhost:11434/v1 (nomic-embed-text, qwen3:4b), OPENAI_TIMEOUT_MS=180000
    • Store: file-based, moderate size (not a large corpus)
    • Full hybrid chain verified healthy immediately before the repro: Ollama /api/tags 200 in 32 ms, /v1/embeddings returns a full 1024-dim vector in <1 s, and POST /agentmemory/search (BM25-only), /lessons/search, GET /sessions all return 200

    Repro (2026-09-17)

    # baseline
    GET /agentmemory/health   -> 200, function_count: 272
    
    # one request
    POST /agentmemory/smart-search {"query":"auth","limit":1}
    -> {"error":"Invocation stopped","error_id":"18b0e53af84c"}   (fails in ~1 s, nowhere near default_timeout 180000)
    
    # immediately after — whole function table gone
    GET /agentmemory/health
    -> {"error":"Function middleware::api-auth not found","error_id":"18b0e6045012"}
    
    # +7 s — fully recovered, function_count 272 again
    

    Additional error_ids from earlier attempts today: 17d25ea28106, 18a60266e165, 17d410ae9522.

    Two observations that might help narrowing:

    1. During the crash window, a request with extra parameters ({"query":"…","limit":2,"types":["memory"]}) returned Function api::smart-search not found instead of Invocation stopped — consistent with the worker being unregistered rather than the function itself failing a second time.
    2. None of the suspected dependencies are at fault here either (same conclusion as the issue body): embedding endpoint, LLM endpoint, BM25 search path all verified healthy against the exact models configured via the OPENAI_* env of the worker process.

    This rules out macOS-specific worker behavior, the bge-m3/deepseek provider combination, and large store size as necessary conditions — on this box the store is modest and models are different, but mem::smart-search still takes the whole worker down the same way. Hoping the fix-shape suggestion in the issue body (contain the failed function instead of deregistering the worker's table) gets picked up; it would make this survivable even if the vector leg keeps throwing.

  5. JCokP commented on Sep 25, 2026

    @JCokP

    Confirming @fatesealed's graph-size explanation: I fixed it by removing the graph.

    Env: 0.9.29, iii 0.11.2, Node v22.23.2, macOS 26.6 arm64, local embeddings, deepseek-v4-flash.

    Graph size: mem:graph:nodes 194 MB (25,260 entries), mem:graph:edges 320 MB (54,788 entries). That is well over the 100 MiB ws limit. Most of the size is sourceObservationIds, which never shrinks: median ~200 ids per node, max 5,175. /graph/stats showed only 563 nodes because it reads the snapshot.

    Fix: I stopped the service, moved the mem%3Agraph%3A*.bin files out of state_store.db/, and restarted.

    before after
    POST /smart-search 500 Invocation stopped, worker drops 200, ~40 ms
    POST /reflect 500 Invocation stopped, worker drops 200, ~13 s

    /graph/reset doesn't help here. It only writes an empty snapshot and leaves the node/edge rows in place.

    Also affected: mem::reflect also runs kv.list on the whole graph. The scheduled reflect had timed out on every run (145 in a row).

    The graph grows back: GRAPH_EXTRACTION_ENABLED=false only gates the LLM pass. event::session::stopped still fires mem::graph-extract unconditionally; one session end after the cleanup added 896 nodes. My local patch:

    -if (compressed.length > 0) fireVoid("mem::graph-extract", { observations: compressed });
    +if (compressed.length > 0 && isGraphExtractionEnabled()) fireVoid("mem::graph-extract", { observations: compressed });

    Suggestions: make graph reads in GraphRetrieval and mem::reflect bounded (see #1300 / #1351), cap sourceObservationIds, and make the flag also disable structural extraction.

  6. rohitg00 commented on Oct 8, 2026

    @rohitg00
    Owner

    @i-sannikov fixed in v0.9.30: bounded graph provenance stops huge rows from crash-looping the worker during search, and AGENTMEMORY_GRAPH_WEIGHT=0 now skips graph retrieval entirely. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions