Repository navigation
Description
Summary
POST /agentmemory/graph/build returns {"error":"Invocation stopped"} to the caller after ~3 minutes, but the server-side batch loop keeps running for hours, steadily burning LLM calls and stalling/crashing the worker. Independently, every bulk or small-batch backfill path hits the same per-batch 180s state::set wall already described in #1127 — so on a large graph store there is currently no feasible way to backfill graph extraction debt.
This corroborates #1127 from a different environment (Windows + remote LLM provider instead of macOS + local Ollama) and adds: (a) the build loop outliving its HTTP failure, (b) a quantified store-size trigger, (c) exact debt measurement via audit records.
Environment
- agentmemory v0.9.29 (latest on npm), iii-engine 0.11.2 (pinned), Windows 10 (10.0.26200), daemon under pm2
- LLM: remote OpenAI-compatible provider (
agnes-2.5-flash), ~12s per extraction call, no rate limiting observed (usedFallback=falsethroughout) GRAPH_EXTRACTION_ENABLED=true; graph store: 8.8k nodes / 15.9k edges (graph:nodes.bin41 MB,graph:edges.bin62 MB); 156 sessions / ~3.6k observations
Repro 1 — graph/build dies to the caller but keeps running server-side
POST /agentmemory/graph/build (empty body, default batchSize 25)
→ ~3 min later: {"error":"Invocation stopped","error_id":"92f87843d444"} (2 batches persisted)
After that HTTP failure, the pm2 log kept emitting, at ~1 line/min, for 7+ hours until the host was rebooted:
[agentmemory] warn graph-build batch failed {"sessionId":"…","batchIndex":0,"error":"Invocation timeout after 180000ms: mem::graph-extract"}
[agentmemory] error Graph extraction failed {"error":"Invocation timeout after 180000ms: state::set"}
During these windows the worker wedges: health.workers[0].latest_metrics freezes, unrelated invocations fail fast with Invocation stopped, and the worker reconnects (fresh uptime_seconds). iii-engine logs show worker re-init noise (failed to initialize worker 'iii-http': address 127.0.0.1:3111 is already in use).
Repro 2 — the per-batch wall is size- and model-independent (matches #1127)
Driving POST /agentmemory/graph/extract directly with small batches from a custom script (pending set computed as titled observations − audit targetIds of mem::graph-extract):
- Batch of 3 / 5 / 8 / 10 observations: LLM call completes in ~12–13s, then
persistGraphDeltawedges atstate::set→Invocation timeout after 180000ms: state::set. Batch size is irrelevant — a 3-obs batch dies identically. - Right after a host reboot the first small batch succeeds (+26 nodes / +34 edges in 11.7s); by the 3rd–4th batch the wedge returns. Consistent with the merge-path write amplification documented in perf(graph): extract only observations captured since the last extract #1268 (3 engine round-trips per node) against the 41–62 MB graph namespaces.
- No
LLM graph extraction failedlines at all — the model always returns; the failure is entirely in the write path.
Consequence — extraction debt accumulates with no recovery path
event::session::stoppedextraction fails chronically at this store size: 11.3% of 2,785 extractions over ~10 days (34% on high-load days).- Debt is measurable: ~1,400 titled observations were never successfully extracted (audit
mem::graph-extract.targetIdsvs titled observations diff). graph/buildcannot complete: a synchronous loop over 156 sessions / ~170 batches needs 1.5–2.5h against a 180s invocation cap, and it restarts from session fix: system audit -- 10 bugs fixed across hooks, triggers, and core #1 on every retry.- Graph quality degrades quietly: search still works (graph weight 0.2), but the traversal layer is chronically under-populated.
Suggestions
- Merge perf(graph): extract only observations captured since the last extract #1268 (incremental
session::stoppedextraction) — it removes the quadratic re-extraction that both inflates daily failures and pre-wedges the worker. - Make
graph/buildresumable and/or async: the audit records (mem::graph-extracttargetIds) already form a natural checkpoint; skipping already-extracted observations would make retries cheap and idempotent. - Consider chunking/pipelining
persistGraphDeltawrites (batchedstate::set, or namespace sharding for large graphs) so a single extraction can't occupy the engine for minutes.
Happy to provide more logs/metrics if useful — thanks for the project!
Activity
sageri commentedon Sep 6, 2026
Proposed concrete fixes (code-level)
I've read through
api::graph-buildandpersistGraphDeltaindist(v0.9.29) while investigating; here are two immediately actionable patches that would unblock backfill on large stores. Happy to turn either into a PR if the approach sounds right.Fix 1 —
api::graph-build: resumable, time-budgeted, self-terminatingThree small changes to the existing loop:
1a. Resume checkpoint — skip already-extracted observations. Successful extractions are already audited with their obs ids, so no new storage is needed:
This turns every retry from "restart from session #1" into "only the remaining debt" (on my store: 169 batches → the ~1400 never-extracted observations), and makes repeated
graph/buildcalls cheap and idempotent — the natural polling loop for a budgeted build.1b. Time budget + truncated response. Check elapsed time between batches and bail out gracefully so a single invocation never approaches the 180s cap:
Together with 1a, callers poll
graph/builduntiltruncated: false.1c. The budget also bounds the zombie loop. In the failure mode from this issue's Repro 1, the HTTP side already returned
Invocation stoppedwhile the loop kept running for hours. With a budget check between batches the loop self-terminates within ~2–3 min even when its own invocation has been stopped. (True cooperative cancellation would need an AbortSignal threaded throughregisterFunctionhandlers — an SDK-level feature, probably worth its own issue.)Fix 2 —
persistGraphDelta: skip no-op mergesOn the merge path (
existingIdfound), a re-extraction of already-processed observations still payskv.setper node/edge — 3 engine round-trips each against multi-MB namespaces — even when the merge carries no new information. Evidence this is the common case: re-running 2 batches over sessions whose extraction had previously succeeded added only +6 nodes / +28 edges, i.e. the vast majority of merge writes change nothing material.where
nodeMergeCarriesChangeis true iff any of: a source obsId not already inexisting.sourceObsIds(or equivalent), a concept/type-weight change, or a degree-affecting field. One semantic decision needed: ifmergeNodebumps alastSeen/timestamp on every call, either (a) keep the write but only when something else changed too, or (b) accept that skipped no-ops don't refreshlastSeen(reasonable for re-extractions, since the obs ids were already recorded). The same guard applies to the edge merge loop.On a 8.8k-node store this removes most of the write volume from re-extractions (
event::session::stoppedcurrently re-sends the whole session every turn — see #1268) and directly reduces the number of longstate::setcalls that trip the 180s timeout.Bigger items (separate from these two patches)
For completeness, the remaining escalations from my earlier comment: keeping
graph:name-index/node-degreeresident in memory, rebuildinggraph:snapshotoff the hot path, pipelining/batching the remainingkv.sets, namespace sharding for 40MB+ graph stores, and merging #1268 (incrementalsession::stoppedextraction) — that last one being the biggest daily-failure-rate win.fix(api): make graph/build resumable and wall-clock bounded (rohitg00…
docs(triage): keep five iii-symptom rows as candidates
docs(triage): keep five iii-symptom rows as candidates
feat: single-flight, cancellable and confirmed graph rebuild