Repository navigation
Description
What happened
A long-running daemon (agentmemory 0.9.29 + iii-engine 0.11.2, file_based kv store) degraded into a wedge: REST worker routes stopped answering while the gateway kept accepting connections. Upgrading to 0.9.30 + the pinned engine 0.22.1 gives a healthy daemon — that cannot read the old store, so there is no way to migrate the 985 memories / 27k-node graph off it.
Windows 11, native engine binaries (no Docker).
1. The old stack wedges; symptoms are gateway-level
Store C:\Users\Leo\data\state_store.db (348 MB, 331 shards), stack 0.9.29 + iii 0.11.2, ~60 min uptime:
| probe | result |
|---|---|
agentmemory status (CLI, direct kv read) |
v0.9.29 · degraded · Sessions 349 · Observations 8719 · Memories 985 · Graph 27256 nodes / 54396 edges · Heap 917 MB |
GET /agentmemory/livez |
200 in 5.9 s |
GET /agentmemory/health |
times out (>8 s, no response) |
GET /agentmemory/sessions |
times out (>8 s, no response) |
netstat |
~10 sockets ESTABLISHED on 3111, never closing |
Consumers time out: an extension hook that awaited smart-search was killed by the harness at 30 s, every prompt. Health went degraded → critical and heap sat at 770–917 MB within an hour of a fresh boot.
2. Engine 0.22.1 does not surface the old store
agentmemory --data-dir C:\Users\Leo\data (0.9.30, pinned engine 0.22.1 installed to ~/.agentmemory/bin/iii.exe):
- engine boots, REST binds ~20 s
- worker log:
Loaded persisted vector index (355 vectors)→ vector index is readable - then:
agentmemory worker did not become ready within 15s.→process.exit(1)(hardcoded 15 s, no flag/env to raise it) - afterwards
statusreportsSessions 0 · Observations 0 · Memories 0 · Graph 0 nodes, 0 edges · Provider ✗ noop (no key)— memories/graph/bm25 keys are invisible, only the vector index loaded - in one run the engine grew to 1.3 GB RSS without ever serving REST
3. No migration surface
agentmemory --help(0.9.30): noexport,backup,migrate, orrestorecommand.- MCP surface (54 tools, enumerated via
tools/listonagentmemory mcp): no import/restore/bulk-write/seed tool —memory_saveinserts one record at a time. memory_exporttakes no arguments and refuses on a real store:
{"bytes":320615835,"error":"Response is 305.8 MiB, over the ~15 MiB limit","limitBytes":15728640,"oversized":true,"success":false}
It also proxies over REST, so on a wedged daemon it is unreachable anyway.
Net: 305.8 MiB of history is readable by the old (wedging) stack only, and there is no supported path to move it.
4. Embeddings regressed across the upgrade
Same ~/.agentmemory/.env: 0.9.29 reports Provider ✓ llm · Embeddings ✓ embeddings; 0.9.30 on a fresh store reports Provider ✓ llm · Embeddings bm25-only. Silent recall-quality downgrade on upgrade; status shows it but nothing warns during upgrade.
Asks
- A documented store migration (0.11.x file_based kv → 0.22.x), or a
migrate --from <dir>command that reads the old shards and replays them. - Make the worker readiness gate non-fatal and configurable (flag/env, default higher than 15 s). Aborting at 15 s on a 348 MB store leaves an engine running with no worker — indistinguishable from the wedge.
- Paginate/filter
memory_export(limit,offset,include: {memories, observations, graph}) so a 300 MB store is exportable under the shim's response cap. - Detect and warn on embedding-provider loss in
status/upgradeoutput.
Workaround applied
Fresh store + pinned 0.22.1: healthy, 34 MB heap, GET /agentmemory/health 5 ms, save→recall round trip verified. History is kept as a cold archive at the old path. Separately, the consumer side was hardened: every extension request now has a bounded timeout (8 s default, 2 s health, 6 s smart-search), so a wedged daemon costs ~10 s per prompt instead of a 30 s harness timeout.
Activity
yoropico commentedon Oct 21, 2026
Confirming the same two blockers on a real 0.9.29 (iii 0.11.2) store copied into 0.9.30 (iii 0.22.1), with measurements:
/livezafter 9-20 s on a 2.9 GB/data(6,977 kv files), so 3 of 4 container starts exited with "agentmemory worker did not become ready within 15s" until fix: allow slow workers to finish startup across CLI paths #1531; with the wait lifted, engine and worker were ready at t+12-15 s.countsshows observations 474,414 / sessions 7,863 butmemories: 0(34,631 on 0.9.29); the firstkv.list("mem:memories")(168 MB scope) drops the worker connection and every function is "not found" until it re-registers. Details and scope sizes in Engine responses over the ws default 100 MiB frame limit tear down the worker connection; state::list has no pagination to bound whole-scope reads #1585./data/iii-config.runtime.yamlbehind; 0.9.30 regenerates it, so that file is not the problem, but the railway template's block-styleallowed_origins(engine-config: block-styleallowed_originsproduces invalid iii-config.runtime.yaml (engine crash-loops; affects deploy/coolify) #1508) still makes the first 0.9.30 boot fail until the list is written flow-style.A documented migration (or an export/import that pages past the frame limit —
/exportreturns 413 on this store) would be needed for stores of this size.rohitg00 commentedon Nov 9, 2026
Fixed in 0.9.33. It pins iii 0.25.1, which serves current stores. The first start installs the matching engine (#1634, #1694). Export and whole-scope reads page past the old 15 MiB limit (#1636). A 21 MB store that returned 500 on 0.9.32 now loads.