Skip to content

mem::graph-extract hangs to its own 180s invocation timeout on every batch, eventually stalling state::get and crashing the worker #1127

Open

Description

@deemaxx

Setup

  • agentmemory: @agentmemory/agentmemory@0.9.27 at time of incident (since upgraded to 0.9.28 during investigation — hang persisted after upgrade based on log/behavior review, not independently re-verified with a fresh 7-day soak)
  • Runtime: macOS (darwin arm64), launched via LaunchAgent (com.agentmemory.server.plist)
  • LLM provider: local Ollama via OpenAI-compatible endpoint (OPENAI_BASE_URL=http://localhost:11434/v1, OPENAI_MODEL=qwen2.5-coder:7b)
  • AGENTMEMORY_LLM_TIMEOUT_MS=120000
  • GRAPH_EXTRACTION_ENABLED=true
  • Hourly cron (com.agentmemory.graph-build.plist, StartInterval: 3600) calling POST /agentmemory/graph/build with {"batchSize":10}

Symptom

Starting ~2026-07-23T19:57Z, every hourly graph/build cron invocation returned an empty response body (curl -s -m 3300 ... timed out or returned nothing) — this repeated roughly every hour for 7 straight days with zero successful runs, versus clean {"batches":N,"edges":N,"nodes":N,"sessions":N,"success":true} results before that point.

Server-side log (~/Library/Logs/agentmemory-server.log) shows the actual failure, repeated across many sessions and restarts:

[agentmemory] warn graph-build batch failed {"sessionId":"<id>","batchIndex":N,"error":"Invocation timeout after 180000ms: mem::graph-extract"}

This fires for many batchIndex values per session, across multiple distinct sessions, across at least 9 separate worker restarts recorded in the log. Confirmed live: POST /agentmemory/graph/build from a plain curl -v hangs and returns zero bytes even after 20s (well under both the 120s LLM timeout and the 180s function-invocation timeout, so this isn't just "slow LLM").

At least once, the accumulated hang escalated further and stalled the graph snapshot read itself:

[agentmemory] warn Graph snapshot read failed {"error":"Invocation timeout after 180000ms: state::get"}

This appears to correlate with full worker crashes/restarts (KeepAlive.SuccessfulExit=false in the plist relaunches the process, which then repeats the same pattern).

Ollama itself was confirmed healthy and responsive throughout (curl .../v1/chat/completions returns 200 with a real completion in ~1-2s), ruling out "LLM provider unreachable" as the cause.

Impact

Because the hourly cron's curl -s silently swallowed the failed/empty response with no alerting, this ran undetected for 7 days. Worse: this appears to have contributed to (though is not confirmed as the sole cause of) a broader persistence gap where no new session data was durably written for the same window — investigated separately, root-caused to an unrelated relative-path config issue (iii-config.yaml's ./data/state_store.db + missing launchd WorkingDirectory), but the graph-extract hang's periodic worker crashes were part of what obscured that second issue for so long.

What I could confirm vs. couldn't

  • Confirmed: mem::graph-extract invocations reliably fail via 180s invocation timeout, not an LLM-side error/4xx.
  • Confirmed: this happens across restarts, across sessions, consistently over multiple days — not a one-off.
  • Not confirmed: the exact internal cause of the hang (I don't have visibility into mem::graph-extract's implementation internals/what it's blocked on — no stack trace or deadlock diagnostic surfaces in the log).
  • Not confirmed: whether this reproduces with a non-Ollama provider (Anthropic/OpenAI direct) — my setup only used local Ollama.

Suggested improvements (independent of root cause)

  1. graph-build-cron.sh (or whatever calls /agentmemory/graph/build) should treat an empty/timeout response as a hard failure and alert loudly, not silently no-op.
  2. Repeated mem::graph-extract timeouts should probably trip a circuit breaker / back off rather than retrying every hour indefinitely against what's evidently a stuck code path.
  3. A hung mem::graph-extract batch shouldn't be able to stall unrelated state::get calls — that coupling turned a contained failure (graph feature degraded) into a full worker crash (session capture degraded too).

Happy to provide the full log excerpt or iii-config.yaml if useful.

Activity

  1. sageri commented on Sep 6, 2026

    @sageri

    Reproduced independently on Windows 10 (agentmemory 0.9.29, iii-engine 0.11.2) with a remote OpenAI-compatible provider — same per-batch state::set 180s timeouts, worker stalls/restarts, and no LLM graph extraction failed lines (the model always returns in ~12s). Additional findings: POST /graph/build keeps looping server-side for hours after its own Invocation stopped HTTP response, and at ~8.8k nodes / 15.9k edges (41/62 MB namespaces) there is no feasible backfill path. Details + repro: #1339

  2. possiblyneal commented on Sep 15, 2026

    @possiblyneal

    Three things in this comment, from a systemd host on 0.9.29:

    The rest of this comment is the evidence: a packet capture across one session end, where the bytes come from, what the open PRs do and do not change, and the three changes that fix it. A patch with all three and tests is ready to open as a PR if wanted.

    Host: systemd user unit, @agentmemory/agentmemory 0.9.29 on Node v22.23.2, Linux x64, iii 0.11.2 with the default file_based adapter, OpenAI-compatible provider. Graph: 4,366 nodes, 8,092 edges. Line numbers are dist/index.mjs of the published 0.9.29 tarball; iii-sdk line numbers are its own dist/index.mjs (0.11.2).

    One session end on the wire

    Session end fires mem::graph-extract (19747), and persistGraphDelta (8652) reads the snapshot with one state::get and writes it back with one state::set. A raw-socket capture on the engine's worker port during one session end, beside the unit journal (2026-09-14, local time):

    17:33:11.818  journal  info Session summarized {...}
    17:33:11      engine -> worker   frame 16,776,861 bytes   (state::get result: the stored snapshot)
    17:33:12      worker -> engine   frame 16,789,204 bytes   (state::set: the merged snapshot)
    17:33:12      engine -> worker   RST, no close frame
    17:33:12.316  journal  [iii] Reconnecting in 783ms (attempt 1)...
    17:33:13.113  journal  [iii] Worker registered with ID: f24e29f7-...
    17:36:12.123  journal  error Graph extraction failed {"error":"Invocation timeout after 180000ms: state::set"}
    

    The engine's frame cap is tungstenite's default max_frame_size, 16 MiB (16,777,216 bytes), which iii 0.11.2 leaves unchanged. The read result fits by 355 bytes; the write does not, the engine's read loop exits, the worker is unregistered, and the socket is reset. The engine logs only [iii] Failed to send message: write ECANCELED. The copy on disk here is 16,776,692 bytes, so each session end lands or drops on which nodes ranked into the top set that run. At the time of the capture the stored copy was last written at 15:55, with 13 session ends and 9 Graph extraction failed timeouts since.

    Why the SDK waits 180 s

    trigger (iii-sdk 360) records the invocation in this.invocations and writes the frame; the only exits are a result frame (664) or the setTimeout at 379-383, set from agentmemory's invocationTimeoutMs: 18e4 (22730). onSocketClose (570-577) terminates the socket and schedules the reconnect without touching this.invocations, and onSocketOpen (578-600) replays only messages queued while disconnected. A frame already written to the dead socket is neither replayed nor failed, so everything in flight at the reset times out 180 s later under its own function's name. In seven days here that was 69 state::set, 5 mem::reflect and 1 state::get, and nothing in the log tells them apart from a slow KV. fireVoid sends mem::graph-extract with no invocation id (iii-sdk 363-372), so the extraction itself is never reported at all, which is the orphaned-invocation note in #1299. This is the coupling the issue's third suggestion asks about.

    Where the bytes come from

    The snapshot is the 500 top-degree nodes, their edges and the stats, and each entry copies the row's full sourceObservationIds. mergeNode (8485) and mergeEdge (8500) union those lists without bound, and the top-degree nodes are the most-mentioned ones, so the snapshot holds the longest lists in the store and grows with every mention:

    part bytes of which sourceObservationIds
    topNodes (500 nodes) 12,953,241 12,746,884
    topEdges (2,014 edges) 3,808,081 3,378,058
    whole snapshot 16,776,692 16,124,942 (96%)

    The largest entry is one file node carrying 6,666 ids. Without the lists the snapshot is about 650 KB. #1168, #1171 and #1360 report the same lists from the query side.

    What the open PRs change, and what they do not

    A failed read becomes an empty snapshot

    readSnapshot (8381) catches every error, logs Graph snapshot read failed (8387) and returns null. persistGraphDelta then does const snap = await readSnapshot(kv) ?? emptySnapshot(); (8653), merges the batch and writes it (8725). A state::get orphaned by the reset, or failed for any other reason, is indistinguishable from no snapshot being stored, so the next delta writes a snapshot holding one batch over the real one. That write is small, so it succeeds, and graph-query answers from the snapshot, so the graph looks collapsed to a few nodes until the next full rebuild.

    What fixes it

    Three changes in src/functions/graph.ts and src/state/kv.ts, in the patch mentioned above with tests (full suite passes):

    1. Write the snapshot without provenance. Snapshot entries get sourceObservationIds: []; the full lists stay on the node and edge rows. The snapshot here goes from 16.8 MB to about 650 KB and stops growing with mentions; a snapshot written before the change is slimmed on its next write. The snapshot page of graph-query then shows empty lists; the depth-walk path reads rows and is unchanged. If PR fix(graph): bound sourceObservationIds in graph-extract and project it out of graph-query #1349's includeSources lands, that page should read the row lists when the flag is set.
    2. Refuse an oversized state::set in StateKV.set, using the payloadByteLength and SAFE_PAYLOAD_BYTES the export path already has, so the next value past the cap fails in its caller with scope, key and byte count instead of taking the socket down under every other call.
    3. Distinguish a failed snapshot read from a missing one in persistGraphDelta. The delta reads through a variant that throws and aborts before writing any row; read-only callers keep the null-as-missing behaviour.

    Outside this repo, the SDK should reject this.invocations in onSocketClose. It is not needed for the three changes above to end the timeouts.

    0006-fix-graph-keep-the-snapshot-under-the-engine-frame-l.patch

  3. rohitg00 commented on Oct 8, 2026

    @rohitg00
    Owner

    @deemaxx partially addressed in v0.9.30: graph provenance is capped at 32 ids and compacted, which removes the row bloat that made every extract batch heavy. Whole-graph scans in extraction are not changed yet. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions