Repository navigation
Description
Current-main boundary
On main c5b8ebf (= v0.9.31), the compose project starts the agentmemory worker with start_after: [state, http]. In the boot below, start_after let the agentmemory worker connect 0.7 s after the state worker connected, but 0.6 s before the engine registered state::get. The worker reads state at once during boot and does not retry function_not_found. Every boot read that loses this race fails once, and its feature stays broken until the next restart.
Reproduction and results (one observed boot)
This happened on the first start of 0.9.31 after an upgrade from 0.9.30 (file state backend). That start also downloaded state 0.22.20 and http 0.22.7. I have seen it once and have not yet measured how often a restart repeats it.
Engine log (compose-state/<id>/agentmemory-3111/engine.log, local time):
[05:28:50.123 PM] [INFO] iii::worker_connections Worker registered <- state and http workers connect
[05:28:50.816 PM] [INFO] iii::worker_connections Worker registered <- agentmemory worker connects
[05:28:50.828 PM] [ERROR] iii::engine Function not found <- function_id: state::get (9 errors up to 05:28:51.092, for state::get/set/list)
[05:28:51.438 PM] [INFO] iii::function [REGISTERED] Function state::get in namespace agentmemory-3111
Worker log (compose-state/<id>/agentmemory-3111/logs/agentmemory.log):
warn index persistence: vector metadata read failed {"message":"function_not_found: Function state::get not found in namespace agentmemory-3111."}
warn slot defaults seed failed {"error":"function_not_found: Function state::get not found in namespace agentmemory-3111."}
warn Vector recovery paused until restart: Persisted vector checkpoint is unavailable
[agentmemory] Failed to rebuild session index at boot: InvocationError: function_not_found: Function state::get not found in namespace agentmemory-3111.
warn rebuildKeywordIndex: failed to load memories {"error":"function_not_found: Function state::list not found in namespace agentmemory-3111."}
GET /agentmemory/status five minutes after the boot:
index.vectorDocuments: 21,pendingVectorBackfill: 22813,vectorBackfillState: "paused". The legacy vector snapshot on disk (about 15k vectors,mem:index:bm25:vec:*) was not loaded, and its migration did not run.index.memoriesIndexed: 0. Saved memories are not in the keyword index, so search returns observations only.- The session index was not rebuilt, and slot defaults were not seeded.
- The status problems point at other causes.
index-save-failingreports a save failure, andindex-vector-backfill-pendingsays "the embedding provider did not return vectors". The provider was healthy (provider.circuitBreaker.state: "closed").
The data on disk is safe: runSave returns while recovery is paused, so the paused state does not overwrite the legacy snapshot.
Required behavior
- Boot does not read state before the state functions exist. Either
start_afterwaits for the dependency's functions, or the worker waits (bounded by the startup timeout) untilstate::getanswers, and retries onfunction_not_found. - A boot read that fails for a temporary reason does not leave vector recovery paused, the keyword index without memories, or the session index stale until a manual restart.
Acceptance checks
In a boot test, the state functions register 2 s after the agentmemory worker connects. The boot ends with the vector checkpoint loaded (or the legacy snapshot migrated), memoriesIndexed equal to the memory count, and no Vector recovery paused warning.
Source and existing work
compose start_after, boot vector load and backfill, metadata read failure returns "unavailable", recovery pauses until restart, keyword rebuild at boot, memories load failure, session index rebuild at boot, slot seed, save gate while paused.
Worker Compose came in #1592. #1594 is a different vector-index stall (the shortfall check). I found no open issue or PR for this race.
Environment: agentmemory 0.9.31 (npm global), iii engine 0.25.0, state worker 0.22.20, http worker 0.22.7, Node v26.12.1, macOS 27.0.1 (arm64).
Activity
Rex57 commentedon Oct 27, 2026
The race happens on every boot here: 3 of 3 on 2026-10-27. The first boot came after the upgrade. The other two were full restarts (the engine log shows all workers unregistered, then compose starting again). The third boot downloaded nothing (
Downloads No downloads), so the first-run worker download is not the cause.Engine log, local time:
Function not foundstate::getregisteredEach boot logged the same failures as in the first report: vector metadata read, slot defaults seed, session index rebuild, and
rebuildKeywordIndex: failed to load memories. In the compose log, the state worker shows✓ state 0.22.20 ready (241ms)right afterwaiting for engine registration. But the engine registersstate::getabout 1.3 s after the state worker connects. On this machine the race is deterministic, so a restart is not a workaround.The embedding provider is healthy, although
index-vector-backfill-pendingblames it. With the same key and model (openai/text-embedding-3-smallthrough OpenRouter), 6 of 6 direct/embeddingscalls returned 1536-dim vectors.vectorDocumentsalso grows for new observations after each boot. Only the documents from before the upgrade stay without vectors, because the paused recovery never loads or migrates the legacy snapshot.Rex57 commentedon Oct 27, 2026
Correction to the report and my comment above: the
mem:index:bm25:vec:*files are not a legacy snapshot. They are the v3 bucket checkpoint from #1416, which 0.9.30 already wrote (vectors:metaplus 47 buckets, about 23k vectors, not 15k). So a lost boot does not skip a migration. It fails to readvectors:metaand never loads the checkpoint. No migration was needed.More evidence that this is the race: restarting only the agentmemory worker, while the state worker stays up, gives a clean boot. I sent
kill -9to the worker (node dist/index.mjs), and compose restarted it after about 11 s (restart: on-failure). That boot logged nofunction_not_found. It loaded the checkpoint (vectorDocuments23,126), indexed all 32 memories and saved the checkpoint again. I did this twice, with the same result. Full starts still lose the race: 4 of 4 now.fix: bound probes by the readiness deadline and fail fast (rohitg00#1623
rosterloh commentedon Oct 28, 2026
Another data point from a self-hosted Docker deployment (0.9.31, iii 0.25.0, file state, ~367 sessions / ~34k observations / ~36k vectors).
Timing varies, and can exceed 15 s. As a stopgap I first delayed the agentmemory worker with
sh -c "sleep 5; exec node dist/index.mjs". One full start booted clean; the next full start lost again (34,381 observations missing from the keyword index, 0 memories indexed, vector recovery paused). I didn't capture engine timestamps for that boot, so I can't give the gap, but together with the ~21 s cold start reported in #1626 it suggests the bounded wait in #1627 (15 s) may be too short for larger stores. Something on the order of #1626's ~70 s, or the composestartup_timeout, seems safer.One more casualty not listed above: a lost boot also left the graph-index migration
paused(sealed, all 4,328 records processed) until I calledPOST /agentmemory/graph/index/resume.External workaround until a fix ships: wait for
state::getbefore starting the worker, in the compose manifest:The namespace must match the compose
namespace:. The first full restart with this in place came up clean: vector checkpoint loaded, all memories indexed, nothing missing from the keyword index.ABrocksdorff commentedon Nov 2, 2026
Another confirmed instance, with timestamps from a Linux box running the packaged Worker Compose
stack (agentmemory 0.9.32-rc.0, iii 0.25.0, state 0.22.20 / http 0.22.7).
Engine log for one cold boot:
01:15:14.715 PM— the agentmemory worker registers its functions01:15:17.450 PM—state::list/state::list_keys/state::list_groupsregister~2.7 s gap, and the current retry window in
runKeywordRebuildis 3 attempts / 250+500 ms, so therebuild gives up long before the state worker appears:
/agentmemory/statusafter that boot:bm25Incompletetrue(headline: "The keyword index could not load every session at boot")bm25Documents/observationsIndexedmissingObservationsmemoriesIndexedindexPersistence.vector.lastErrorSo the same race costs three things at once: keyword recall, memory recall, and the persisted vector
checkpoint — not just the BM25 rebuild.
We worked around it locally with a minimal non-fatal wait before the first boot read (poll
state::list, warn on timeout rather than abort):bm25Documents325 -> 8229,missingObservations7440 -> 0,memoriesIndexed0 -> 157, and the vector checkpoint error went away.We are not opening a PR for it — that is essentially a narrower version of what #1637 already does,
and #1637 also re-arms index and capture recovery, which our hack does not. +1 for landing #1637's
waitForStateWorkerahead of #1626/#1627, and the non-fatal-on-timeout behaviour matters: on this box2.7 s is well inside 60 s, but a fatal probe is what took the packed-artifact install gate down in
#1627.
fix: wait for state worker functions before boot reads (rohitg00#1627)