Skip to content

Worker Compose boot race: first state reads fail with state::get not found, and vector recovery stays paused until restart #1623

Description

@Rex57

Current-main boundary

On main c5b8ebf (= v0.9.31), the compose project starts the agentmemory worker with start_after: [state, http]. In the boot below, start_after let the agentmemory worker connect 0.7 s after the state worker connected, but 0.6 s before the engine registered state::get. The worker reads state at once during boot and does not retry function_not_found. Every boot read that loses this race fails once, and its feature stays broken until the next restart.

Reproduction and results (one observed boot)

This happened on the first start of 0.9.31 after an upgrade from 0.9.30 (file state backend). That start also downloaded state 0.22.20 and http 0.22.7. I have seen it once and have not yet measured how often a restart repeats it.

Engine log (compose-state/<id>/agentmemory-3111/engine.log, local time):

[05:28:50.123 PM] [INFO] iii::worker_connections Worker registered        <- state and http workers connect
[05:28:50.816 PM] [INFO] iii::worker_connections Worker registered        <- agentmemory worker connects
[05:28:50.828 PM] [ERROR] iii::engine Function not found                  <- function_id: state::get (9 errors up to 05:28:51.092, for state::get/set/list)
[05:28:51.438 PM] [INFO] iii::function [REGISTERED] Function state::get in namespace agentmemory-3111

Worker log (compose-state/<id>/agentmemory-3111/logs/agentmemory.log):

warn index persistence: vector metadata read failed {"message":"function_not_found: Function state::get not found in namespace agentmemory-3111."}
warn slot defaults seed failed {"error":"function_not_found: Function state::get not found in namespace agentmemory-3111."}
warn Vector recovery paused until restart: Persisted vector checkpoint is unavailable
[agentmemory] Failed to rebuild session index at boot: InvocationError: function_not_found: Function state::get not found in namespace agentmemory-3111.
warn rebuildKeywordIndex: failed to load memories {"error":"function_not_found: Function state::list not found in namespace agentmemory-3111."}

GET /agentmemory/status five minutes after the boot:

  1. index.vectorDocuments: 21, pendingVectorBackfill: 22813, vectorBackfillState: "paused". The legacy vector snapshot on disk (about 15k vectors, mem:index:bm25:vec:*) was not loaded, and its migration did not run.
  2. index.memoriesIndexed: 0. Saved memories are not in the keyword index, so search returns observations only.
  3. The session index was not rebuilt, and slot defaults were not seeded.
  4. The status problems point at other causes. index-save-failing reports a save failure, and index-vector-backfill-pending says "the embedding provider did not return vectors". The provider was healthy (provider.circuitBreaker.state: "closed").

The data on disk is safe: runSave returns while recovery is paused, so the paused state does not overwrite the legacy snapshot.

Required behavior

  • Boot does not read state before the state functions exist. Either start_after waits for the dependency's functions, or the worker waits (bounded by the startup timeout) until state::get answers, and retries on function_not_found.
  • A boot read that fails for a temporary reason does not leave vector recovery paused, the keyword index without memories, or the session index stale until a manual restart.

Acceptance checks

In a boot test, the state functions register 2 s after the agentmemory worker connects. The boot ends with the vector checkpoint loaded (or the legacy snapshot migrated), memoriesIndexed equal to the memory count, and no Vector recovery paused warning.

Source and existing work

compose start_after, boot vector load and backfill, metadata read failure returns "unavailable", recovery pauses until restart, keyword rebuild at boot, memories load failure, session index rebuild at boot, slot seed, save gate while paused.

Worker Compose came in #1592. #1594 is a different vector-index stall (the shortfall check). I found no open issue or PR for this race.

Environment: agentmemory 0.9.31 (npm global), iii engine 0.25.0, state worker 0.22.20, http worker 0.22.7, Node v26.12.1, macOS 27.0.1 (arm64).

Activity

  1. Rex57 commented on Oct 27, 2026

    @Rex57
    ContributorAuthor

    The race happens on every boot here: 3 of 3 on 2026-10-27. The first boot came after the upgrade. The other two were full restarts (the engine log shows all workers unregistered, then compose starting again). The third boot downloaded nothing (Downloads No downloads), so the first-run worker download is not the cause.

    Engine log, local time:

    Boot agentmemory worker connects Function not found state::get registered Gap
    1 (17:28) 50.816 9, from 50.828 to 51.092 51.438 0.62 s
    2 (17:40) 52.485 9, from 52.497 to 52.756 53.207 0.72 s
    3 (17:45) 26.110 10, from 26.121 to 26.672 26.879 0.77 s

    Each boot logged the same failures as in the first report: vector metadata read, slot defaults seed, session index rebuild, and rebuildKeywordIndex: failed to load memories. In the compose log, the state worker shows ✓ state 0.22.20 ready (241ms) right after waiting for engine registration. But the engine registers state::get about 1.3 s after the state worker connects. On this machine the race is deterministic, so a restart is not a workaround.

    The embedding provider is healthy, although index-vector-backfill-pending blames it. With the same key and model (openai/text-embedding-3-small through OpenRouter), 6 of 6 direct /embeddings calls returned 1536-dim vectors. vectorDocuments also grows for new observations after each boot. Only the documents from before the upgrade stay without vectors, because the paused recovery never loads or migrates the legacy snapshot.

  2. Rex57 commented on Oct 27, 2026

    @Rex57
    ContributorAuthor

    Correction to the report and my comment above: the mem:index:bm25:vec:* files are not a legacy snapshot. They are the v3 bucket checkpoint from #1416, which 0.9.30 already wrote (vectors:meta plus 47 buckets, about 23k vectors, not 15k). So a lost boot does not skip a migration. It fails to read vectors:meta and never loads the checkpoint. No migration was needed.

    More evidence that this is the race: restarting only the agentmemory worker, while the state worker stays up, gives a clean boot. I sent kill -9 to the worker (node dist/index.mjs), and compose restarted it after about 11 s (restart: on-failure). That boot logged no function_not_found. It loaded the checkpoint (vectorDocuments 23,126), indexed all 32 memories and saved the checkpoint again. I did this twice, with the same result. Full starts still lose the race: 4 of 4 now.

  3. rosterloh commented on Oct 28, 2026

    @rosterloh

    Another data point from a self-hosted Docker deployment (0.9.31, iii 0.25.0, file state, ~367 sessions / ~34k observations / ~36k vectors).

    Timing varies, and can exceed 15 s. As a stopgap I first delayed the agentmemory worker with sh -c "sleep 5; exec node dist/index.mjs". One full start booted clean; the next full start lost again (34,381 observations missing from the keyword index, 0 memories indexed, vector recovery paused). I didn't capture engine timestamps for that boot, so I can't give the gap, but together with the ~21 s cold start reported in #1626 it suggests the bounded wait in #1627 (15 s) may be too short for larger stores. Something on the order of #1626's ~70 s, or the compose startup_timeout, seems safer.

    One more casualty not listed above: a lost boot also left the graph-index migration paused (sealed, all 4,328 records processed) until I called POST /agentmemory/graph/index/resume.

    External workaround until a fix ships: wait for state::get before starting the worker, in the compose manifest:

    scripts:
      run: sh -c "node /opt/agentmemory/wait-state.mjs; exec node dist/index.mjs"
    // wait-state.mjs
    const { registerWorker } = await import("/opt/agentmemory/node_modules/iii-sdk/dist/index.mjs");
    const sdk = registerWorker("ws://127.0.0.1:49134", { workerName: "state-preflight", namespace: "agentmemory" });
    const deadline = Date.now() + 120_000;
    for (;;) {
      try {
        await sdk.trigger({ function_id: "state::get", payload: { scope: "mem:metrics", key: "__preflight__" } });
        break;
      } catch (err) {
        if (err?.code !== "function_not_found" || Date.now() > deadline) break;
        await new Promise((r) => setTimeout(r, 500));
      }
    }
    process.exit(0);

    The namespace must match the compose namespace:. The first full restart with this in place came up clean: vector checkpoint loaded, all memories indexed, nothing missing from the keyword index.

  4. ABrocksdorff commented on Nov 2, 2026

    @ABrocksdorff

    Another confirmed instance, with timestamps from a Linux box running the packaged Worker Compose
    stack (agentmemory 0.9.32-rc.0, iii 0.25.0, state 0.22.20 / http 0.22.7).

    Engine log for one cold boot:

    • 01:15:14.715 PM — the agentmemory worker registers its functions
    • 01:15:17.450 PM — state::list / state::list_keys / state::list_groups register

    ~2.7 s gap, and the current retry window in runKeywordRebuild is 3 attempts / 250+500 ms, so the
    rebuild gives up long before the state worker appears:

    [agentmemory] warn rebuildKeywordIndex: failed to load memories {"error":"function_not_found: Function state::list not found in namespace agentmemory-3111."}
    [agentmemory] warn rebuildKeywordIndex: failed to load sessions after retries {"attempts":3,"error":"function_not_found: Function state::list not found in namespace agentmemory-3111."}
    

    /agentmemory/status after that boot:

    value
    bm25Incomplete true (headline: "The keyword index could not load every session at boot")
    bm25Documents / observationsIndexed 325
    missingObservations 7440
    memoriesIndexed 0
    indexPersistence.vector.lastError "Vector recovery paused until restart: Persisted vector checkpoint is unavailable"

    So the same race costs three things at once: keyword recall, memory recall, and the persisted vector
    checkpoint — not just the BM25 rebuild.

    We worked around it locally with a minimal non-fatal wait before the first boot read (poll
    state::list, warn on timeout rather than abort): bm25Documents 325 -> 8229,
    missingObservations 7440 -> 0, memoriesIndexed 0 -> 157, and the vector checkpoint error went away.

    We are not opening a PR for it — that is essentially a narrower version of what #1637 already does,
    and #1637 also re-arms index and capture recovery, which our hack does not. +1 for landing #1637's
    waitForStateWorker ahead of #1626/#1627, and the non-fatal-on-timeout behaviour matters: on this box
    2.7 s is well inside 60 s, but a fatal probe is what took the packed-artifact install gate down in
    #1627.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions