Skip to content

graph/build: server-side batch loop outlives its "Invocation stopped" HTTP response; bulk backfill impossible on large graph stores (Win32 + remote LLM, corroborates #1127) #1339

Open

Description

@sageri

Summary

POST /agentmemory/graph/build returns {"error":"Invocation stopped"} to the caller after ~3 minutes, but the server-side batch loop keeps running for hours, steadily burning LLM calls and stalling/crashing the worker. Independently, every bulk or small-batch backfill path hits the same per-batch 180s state::set wall already described in #1127 — so on a large graph store there is currently no feasible way to backfill graph extraction debt.

This corroborates #1127 from a different environment (Windows + remote LLM provider instead of macOS + local Ollama) and adds: (a) the build loop outliving its HTTP failure, (b) a quantified store-size trigger, (c) exact debt measurement via audit records.

Environment

  • agentmemory v0.9.29 (latest on npm), iii-engine 0.11.2 (pinned), Windows 10 (10.0.26200), daemon under pm2
  • LLM: remote OpenAI-compatible provider (agnes-2.5-flash), ~12s per extraction call, no rate limiting observed (usedFallback=false throughout)
  • GRAPH_EXTRACTION_ENABLED=true; graph store: 8.8k nodes / 15.9k edges (graph:nodes.bin 41 MB, graph:edges.bin 62 MB); 156 sessions / ~3.6k observations

Repro 1 — graph/build dies to the caller but keeps running server-side

POST /agentmemory/graph/build   (empty body, default batchSize 25)
→ ~3 min later: {"error":"Invocation stopped","error_id":"92f87843d444"}   (2 batches persisted)

After that HTTP failure, the pm2 log kept emitting, at ~1 line/min, for 7+ hours until the host was rebooted:

[agentmemory] warn graph-build batch failed {"sessionId":"…","batchIndex":0,"error":"Invocation timeout after 180000ms: mem::graph-extract"}
[agentmemory] error Graph extraction failed {"error":"Invocation timeout after 180000ms: state::set"}

During these windows the worker wedges: health.workers[0].latest_metrics freezes, unrelated invocations fail fast with Invocation stopped, and the worker reconnects (fresh uptime_seconds). iii-engine logs show worker re-init noise (failed to initialize worker 'iii-http': address 127.0.0.1:3111 is already in use).

Repro 2 — the per-batch wall is size- and model-independent (matches #1127)

Driving POST /agentmemory/graph/extract directly with small batches from a custom script (pending set computed as titled observations − audit targetIds of mem::graph-extract):

  • Batch of 3 / 5 / 8 / 10 observations: LLM call completes in ~12–13s, then persistGraphDelta wedges at state::set → Invocation timeout after 180000ms: state::set. Batch size is irrelevant — a 3-obs batch dies identically.
  • Right after a host reboot the first small batch succeeds (+26 nodes / +34 edges in 11.7s); by the 3rd–4th batch the wedge returns. Consistent with the merge-path write amplification documented in perf(graph): extract only observations captured since the last extract #1268 (3 engine round-trips per node) against the 41–62 MB graph namespaces.
  • No LLM graph extraction failed lines at all — the model always returns; the failure is entirely in the write path.

Consequence — extraction debt accumulates with no recovery path

  • event::session::stopped extraction fails chronically at this store size: 11.3% of 2,785 extractions over ~10 days (34% on high-load days).
  • Debt is measurable: ~1,400 titled observations were never successfully extracted (audit mem::graph-extract.targetIds vs titled observations diff).
  • graph/build cannot complete: a synchronous loop over 156 sessions / ~170 batches needs 1.5–2.5h against a 180s invocation cap, and it restarts from session fix: system audit -- 10 bugs fixed across hooks, triggers, and core #1 on every retry.
  • Graph quality degrades quietly: search still works (graph weight 0.2), but the traversal layer is chronically under-populated.

Suggestions

  1. Merge perf(graph): extract only observations captured since the last extract #1268 (incremental session::stopped extraction) — it removes the quadratic re-extraction that both inflates daily failures and pre-wedges the worker.
  2. Make graph/build resumable and/or async: the audit records (mem::graph-extract targetIds) already form a natural checkpoint; skipping already-extracted observations would make retries cheap and idempotent.
  3. Consider chunking/pipelining persistGraphDelta writes (batched state::set, or namespace sharding for large graphs) so a single extraction can't occupy the engine for minutes.

Happy to provide more logs/metrics if useful — thanks for the project!

Activity

  1. sageri commented on Sep 6, 2026

    @sageri
    Author

    Proposed concrete fixes (code-level)

    I've read through api::graph-build and persistGraphDelta in dist (v0.9.29) while investigating; here are two immediately actionable patches that would unblock backfill on large stores. Happy to turn either into a PR if the approach sounds right.

    Fix 1 — api::graph-build: resumable, time-budgeted, self-terminating

    Three small changes to the existing loop:

    1a. Resume checkpoint — skip already-extracted observations. Successful extractions are already audited with their obs ids, so no new storage is needed:

    // before the session loop
    const done = new Set<string>();
    for (const e of await kv.list(KV.audit)) {
      if (e.functionId === "mem::graph-extract")
        for (const id of e.targetIds ?? []) done.add(id);
    }
    // inside the loop, replace the plain title filter:
    const compressed = (await kv.list(KV.observations(sid)))
      .filter((o) => o.title)
      .filter((o) => !done.has(o.id));   // only the debt

    This turns every retry from "restart from session #1" into "only the remaining debt" (on my store: 169 batches → the ~1400 never-extracted observations), and makes repeated graph/build calls cheap and idempotent — the natural polling loop for a budgeted build.

    1b. Time budget + truncated response. Check elapsed time between batches and bail out gracefully so a single invocation never approaches the 180s cap:

    const budgetMs = Math.min(Number(req.body?.timeBudgetMs) || 120_000, 150_000);
    const start = Date.now();
    // in the batch loop:
    if (Date.now() - start > budgetMs) { truncated = true; break; }
    // response gains:
    { success: true, sessions, batches, nodes, edges, truncated, remainingSessions, remainingBatches }

    Together with 1a, callers poll graph/build until truncated: false.

    1c. The budget also bounds the zombie loop. In the failure mode from this issue's Repro 1, the HTTP side already returned Invocation stopped while the loop kept running for hours. With a budget check between batches the loop self-terminates within ~2–3 min even when its own invocation has been stopped. (True cooperative cancellation would need an AbortSignal threaded through registerFunction handlers — an SDK-level feature, probably worth its own issue.)

    Fix 2 — persistGraphDelta: skip no-op merges

    On the merge path (existingId found), a re-extraction of already-processed observations still pays kv.set per node/edge — 3 engine round-trips each against multi-MB namespaces — even when the merge carries no new information. Evidence this is the common case: re-running 2 batches over sessions whose extraction had previously succeeded added only +6 nodes / +28 edges, i.e. the vast majority of merge writes change nothing material.

    if (existing) {
      idRemap.set(node.id, existing.id);
      const merged = mergeNode(existing, node, obsIds, capturedAt);
      if (!nodeMergeCarriesChange(existing, merged, obsIds)) continue;  // skip kv.set
      await kv.set(KV.graphNodes, existing.id, merged);
      ...
    }

    where nodeMergeCarriesChange is true iff any of: a source obsId not already in existing.sourceObsIds(or equivalent), a concept/type-weight change, or a degree-affecting field. One semantic decision needed: if mergeNode bumps a lastSeen/timestamp on every call, either (a) keep the write but only when something else changed too, or (b) accept that skipped no-ops don't refresh lastSeen (reasonable for re-extractions, since the obs ids were already recorded). The same guard applies to the edge merge loop.

    On a 8.8k-node store this removes most of the write volume from re-extractions (event::session::stopped currently re-sends the whole session every turn — see #1268) and directly reduces the number of long state::set calls that trip the 180s timeout.

    Bigger items (separate from these two patches)

    For completeness, the remaining escalations from my earlier comment: keeping graph:name-index / node-degree resident in memory, rebuilding graph:snapshot off the hot path, pipelining/batching the remaining kv.sets, namespace sharding for 40MB+ graph stores, and merging #1268 (incremental session::stopped extraction) — that last one being the biggest daily-failure-rate win.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions