Repository navigation
Description
Setup
- agentmemory:
@agentmemory/agentmemory@0.9.27at time of incident (since upgraded to 0.9.28 during investigation — hang persisted after upgrade based on log/behavior review, not independently re-verified with a fresh 7-day soak) - Runtime: macOS (darwin arm64), launched via LaunchAgent (
com.agentmemory.server.plist) - LLM provider: local Ollama via OpenAI-compatible endpoint (
OPENAI_BASE_URL=http://localhost:11434/v1,OPENAI_MODEL=qwen2.5-coder:7b) AGENTMEMORY_LLM_TIMEOUT_MS=120000GRAPH_EXTRACTION_ENABLED=true- Hourly cron (
com.agentmemory.graph-build.plist,StartInterval: 3600) callingPOST /agentmemory/graph/buildwith{"batchSize":10}
Symptom
Starting ~2026-07-23T19:57Z, every hourly graph/build cron invocation returned an empty response body (curl -s -m 3300 ... timed out or returned nothing) — this repeated roughly every hour for 7 straight days with zero successful runs, versus clean {"batches":N,"edges":N,"nodes":N,"sessions":N,"success":true} results before that point.
Server-side log (~/Library/Logs/agentmemory-server.log) shows the actual failure, repeated across many sessions and restarts:
[agentmemory] warn graph-build batch failed {"sessionId":"<id>","batchIndex":N,"error":"Invocation timeout after 180000ms: mem::graph-extract"}
This fires for many batchIndex values per session, across multiple distinct sessions, across at least 9 separate worker restarts recorded in the log. Confirmed live: POST /agentmemory/graph/build from a plain curl -v hangs and returns zero bytes even after 20s (well under both the 120s LLM timeout and the 180s function-invocation timeout, so this isn't just "slow LLM").
At least once, the accumulated hang escalated further and stalled the graph snapshot read itself:
[agentmemory] warn Graph snapshot read failed {"error":"Invocation timeout after 180000ms: state::get"}
This appears to correlate with full worker crashes/restarts (KeepAlive.SuccessfulExit=false in the plist relaunches the process, which then repeats the same pattern).
Ollama itself was confirmed healthy and responsive throughout (curl .../v1/chat/completions returns 200 with a real completion in ~1-2s), ruling out "LLM provider unreachable" as the cause.
Impact
Because the hourly cron's curl -s silently swallowed the failed/empty response with no alerting, this ran undetected for 7 days. Worse: this appears to have contributed to (though is not confirmed as the sole cause of) a broader persistence gap where no new session data was durably written for the same window — investigated separately, root-caused to an unrelated relative-path config issue (iii-config.yaml's ./data/state_store.db + missing launchd WorkingDirectory), but the graph-extract hang's periodic worker crashes were part of what obscured that second issue for so long.
What I could confirm vs. couldn't
- Confirmed:
mem::graph-extractinvocations reliably fail via 180s invocation timeout, not an LLM-side error/4xx. - Confirmed: this happens across restarts, across sessions, consistently over multiple days — not a one-off.
- Not confirmed: the exact internal cause of the hang (I don't have visibility into
mem::graph-extract's implementation internals/what it's blocked on — no stack trace or deadlock diagnostic surfaces in the log). - Not confirmed: whether this reproduces with a non-Ollama provider (Anthropic/OpenAI direct) — my setup only used local Ollama.
Suggested improvements (independent of root cause)
graph-build-cron.sh(or whatever calls/agentmemory/graph/build) should treat an empty/timeout response as a hard failure and alert loudly, not silently no-op.- Repeated
mem::graph-extracttimeouts should probably trip a circuit breaker / back off rather than retrying every hour indefinitely against what's evidently a stuck code path. - A hung
mem::graph-extractbatch shouldn't be able to stall unrelatedstate::getcalls — that coupling turned a contained failure (graph feature degraded) into a full worker crash (session capture degraded too).
Happy to provide the full log excerpt or iii-config.yaml if useful.
Activity
sageri commentedon Sep 6, 2026
Reproduced independently on Windows 10 (agentmemory 0.9.29, iii-engine 0.11.2) with a remote OpenAI-compatible provider — same per-batch
state::set180s timeouts, worker stalls/restarts, and noLLM graph extraction failedlines (the model always returns in ~12s). Additional findings:POST /graph/buildkeeps looping server-side for hours after its ownInvocation stoppedHTTP response, and at ~8.8k nodes / 15.9k edges (41/62 MB namespaces) there is no feasible backfill path. Details + repro: #1339possiblyneal commentedon Sep 15, 2026
Three things in this comment, from a systemd host on 0.9.29:
Invocation timeout after 180000msonstate::set, onstate::get(theGraph snapshot read failedline quoted above) and onmem::reflect, each at exactly 180 s, on a store whose KV round trips otherwise take milliseconds. 61 of the 75 such timeouts in the last seven days land 180 s after a[iii] Reconnectingline.state::setoutgrew the engine's 16 MiB WebSocket frame cap. The engine answers an oversized frame by resetting the worker socket with no close frame, and the SDK reconnects without failing anything in flight, so every call waiting on that socket, whatever function it was for, waits the full 180 s. It is a size limit reported as a timeout, and it recurs at every session end once the snapshot passes the line.readSnapshotturns a failedstate::getinto "no snapshot stored", andpersistGraphDeltathen starts fromemptySnapshot()and writes that over the stored snapshot with the stats and top set of one batch.The rest of this comment is the evidence: a packet capture across one session end, where the bytes come from, what the open PRs do and do not change, and the three changes that fix it. A patch with all three and tests is ready to open as a PR if wanted.
Host: systemd user unit,
@agentmemory/agentmemory0.9.29 on Node v22.23.2, Linux x64, iii 0.11.2 with the defaultfile_basedadapter, OpenAI-compatible provider. Graph: 4,366 nodes, 8,092 edges. Line numbers aredist/index.mjsof the published 0.9.29 tarball;iii-sdkline numbers are its owndist/index.mjs(0.11.2).One session end on the wire
Session end fires
mem::graph-extract(19747), andpersistGraphDelta(8652) reads the snapshot with onestate::getand writes it back with onestate::set. A raw-socket capture on the engine's worker port during one session end, beside the unit journal (2026-09-14, local time):The engine's frame cap is tungstenite's default
max_frame_size, 16 MiB (16,777,216 bytes), which iii 0.11.2 leaves unchanged. The read result fits by 355 bytes; the write does not, the engine's read loop exits, the worker is unregistered, and the socket is reset. The engine logs only[iii] Failed to send message: write ECANCELED. The copy on disk here is 16,776,692 bytes, so each session end lands or drops on which nodes ranked into the top set that run. At the time of the capture the stored copy was last written at 15:55, with 13 session ends and 9Graph extraction failedtimeouts since.Why the SDK waits 180 s
trigger(iii-sdk 360) records the invocation inthis.invocationsand writes the frame; the only exits are a result frame (664) or thesetTimeoutat 379-383, set from agentmemory'sinvocationTimeoutMs: 18e4(22730).onSocketClose(570-577) terminates the socket and schedules the reconnect without touchingthis.invocations, andonSocketOpen(578-600) replays only messages queued while disconnected. A frame already written to the dead socket is neither replayed nor failed, so everything in flight at the reset times out 180 s later under its own function's name. In seven days here that was 69state::set, 5mem::reflectand 1state::get, and nothing in the log tells them apart from a slow KV.fireVoidsendsmem::graph-extractwith no invocation id (iii-sdk 363-372), so the extraction itself is never reported at all, which is the orphaned-invocation note in #1299. This is the coupling the issue's third suggestion asks about.Where the bytes come from
The snapshot is the 500 top-degree nodes, their edges and the stats, and each entry copies the row's full
sourceObservationIds.mergeNode(8485) andmergeEdge(8500) union those lists without bound, and the top-degree nodes are the most-mentioned ones, so the snapshot holds the longest lists in the store and grows with every mention:sourceObservationIdstopNodes(500 nodes)topEdges(2,014 edges)The largest entry is one
filenode carrying 6,666 ids. Without the lists the snapshot is about 650 KB. #1168, #1171 and #1360 report the same lists from the query side.What the open PRs change, and what they do not
timeoutMsonStateKV): the frame is still dropped and the socket still reset.state::setreports at 10 s naming scope and key, which is worth having, but every non-KV call still waits 180 s, the snapshot still never writes, and the overwrite below fires at 10 s instead of 180.GRAPH_MAX_SOURCE_IDS, default 10): caps a row the next time it is merged. The snapshot copies rows and an unmentioned row is never rewritten, so a store already over the line stays there until every top node is touched again. The cap also lands on the data rows, where the full list is the useful part, not on the render cache.checkPayloadFrameSize(7385) for it, called at export and mesh-export only.StateKV.set(1337) sends the value with no size check.A failed read becomes an empty snapshot
readSnapshot(8381) catches every error, logsGraph snapshot read failed(8387) and returns null.persistGraphDeltathen doesconst snap = await readSnapshot(kv) ?? emptySnapshot();(8653), merges the batch and writes it (8725). Astate::getorphaned by the reset, or failed for any other reason, is indistinguishable from no snapshot being stored, so the next delta writes a snapshot holding one batch over the real one. That write is small, so it succeeds, andgraph-queryanswers from the snapshot, so the graph looks collapsed to a few nodes until the next full rebuild.What fixes it
Three changes in
src/functions/graph.tsandsrc/state/kv.ts, in the patch mentioned above with tests (full suite passes):sourceObservationIds: []; the full lists stay on the node and edge rows. The snapshot here goes from 16.8 MB to about 650 KB and stops growing with mentions; a snapshot written before the change is slimmed on its next write. The snapshot page ofgraph-querythen shows empty lists; the depth-walk path reads rows and is unchanged. If PR fix(graph): bound sourceObservationIds in graph-extract and project it out of graph-query #1349'sincludeSourceslands, that page should read the row lists when the flag is set.state::setinStateKV.set, using thepayloadByteLengthandSAFE_PAYLOAD_BYTESthe export path already has, so the next value past the cap fails in its caller with scope, key and byte count instead of taking the socket down under every other call.persistGraphDelta. The delta reads through a variant that throws and aborts before writing any row; read-only callers keep the null-as-missing behaviour.Outside this repo, the SDK should reject
this.invocationsinonSocketClose. It is not needed for the three changes above to end the timeouts.0006-fix-graph-keep-the-snapshot-under-the-engine-frame-l.patch
rohitg00 commentedon Oct 8, 2026
@deemaxx partially addressed in v0.9.30: graph provenance is capped at 32 ids and compacted, which removes the row bloat that made every extract batch heavy. Whole-graph scans in extraction are not changed yet. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30