Skip to content

Shutdown index save is lost to a container teardown race, and boot only reconciles a completely empty index #1335

Open

Description

@herman925

Summary

Two defects that compound. Together they can leave everything indexed since the last restart permanently unsearchable, while the data itself is intact in KV and every exit code says success.

  1. The shutdown save is issued and acknowledged, then lost in transit. save() completes and process.exit(0) is reached ~0.3 s later, while the iii engine needs ~2 s to flush its file-based KV to .bin. Under docker stop, the engine is a child of PID 1 and is SIGKILLed along with the container before the flush lands.
  2. Boot only reconciles a completely empty index. rebuildIndex(kv) runs solely inside if (idx.size === 0). An index that loads even one stale entry has size > 0, so the rebuild is suppressed and nothing repairs the delta.

Neither is visible: exit code 0, no error in the log, and the search API returns confident results from an index that is missing recent records.

Reproduced on a clean install

Stock @agentmemory/agentmemory@0.9.29 on node:22-bookworm-slim with the official iii v0.11.2 binary. Fresh empty volume. No secret, no EMBEDDING_PROVIDER, no LLM key, no slots, no consolidation, no graph extraction — the log confirms No LLM provider key set — running zero-LLM (BM25 + on-device embeddings).

1. Store a memory                → mem_…91ab656701eb at 17:37:20
   Search it                     → 1 result, score 2.9124
2. Shard generation before stop  → mem%3A…%3Aidx_mtn8i4m7_ffc91ded8556%3A00000.bin, mtime 17:35
3. docker stop -t 60             → 0.3 s wall, docker exit 0, container ExitCode 0
                                    log ends at "[agentmemory] Shutting down..."
4. Shard generation after stop   → IDENTICAL, mtime still 17:35

saveShardedIndex mints a new generation on every call and has no skip-if-unchanged path, so an unchanged generation is proof the write did not land.

The save works — it is a teardown race

Running the worker as a non-PID-1 process so the engine outlives it, then sending SIGTERM to the worker alone:

gen BEFORE kill : …idx_mtn8nqb5_f1dfe47fd6b2:00000.bin
worker pid      : 130
NEW GENERATION after ~2 s : …idx_mtn8op40_6301bd8505ad:00000.bin

A/B with one variable, two runs each:

Engine lifetime after worker SIGTERM New generation written
Killed with the container (docker stop) No
Survives the worker Yes, after ~2 s

The process tree is PID 1 = node agentmemory with iii as its child. docker stop signals PID 1 only. So the write is lost in transit, not skipped — which is why no error surfaces anywhere.

The reconcile gate

rebuildIndex(kv) fires only under if (idx.size === 0) (dist/src-G7yt8gGm.mjs:3113). It has no entry cap, no time limit and no batch ceiling — it is simply gated on the index being entirely empty.

The consequence is that reconciliation is all-or-nothing. A partial gap is exactly the case it cannot repair, because a partially-stale index still has size > 0. So the state produced by defect 1 is the one state defect 2 will not fix, and it persists across every subsequent restart.

The only recovery we found is deleting the persisted shards to force size === 0:

docker stop -t 60 <container>
docker run --rm -v <volume>:/d alpine sh -c 'rm -f /d/state_store.db/mem%3Aindex%3Abm25*.bin'
docker start <container>

On the affected store this took a sample of observations from 0/4 retrievable to 4/4, writing a new generation. Worth noting that this works because it forces the empty-index precondition, not because it clears corruption.

Why the loss window is a whole session

DEBOUNCE_MS = 5e3 appears never to fire during runtime. Four memories plus 60 s of idling produced no BM25 shard at all; every generation observed was minted at boot or at a shutdown that was allowed to complete.

If that is right, boot and shutdown are the only durable write paths — and shutdown is the racing one. That is what turns this from "you lose the last five seconds" into "you lose everything since the last restart".

Two things that make it silent

  • Exit code 0 is not evidence. The shutdown handler ends in process.exit(0) regardless of whether the flush landed.
  • Failed to save index on shutdown can never fire. save() wraps its body in try/catch and routes errors to logFailure, so it never rejects and the .catch() at the shutdown call site is unreachable. Not the cause here, but it means a genuine save failure would also be invisible.

Suggested fixes

  1. Await the engine flush before exiting, or give the engine its own SIGTERM and a grace window, rather than reaching process.exit(0) 0.3 s after issuing the save.
  2. Let boot reconcile a partial index, not only an empty one — compare the index against KV rather than gating on size === 0. A partial gap is the realistic failure mode and currently the one case that never repairs.
  3. Make the debounce actually persist during runtime, so a lost shutdown costs seconds rather than a whole session.
  4. Remove or fix the unreachable .catch, so save failures are observable.

Item 1 alone would close the common case.

One thing we could not explain

In one clean-room run a stale shard (mtime 17:35) was present, no new generation was written on restart, and yet the memory stored at 17:37:20 was still found afterwards at the identical pre-stop score. Under the gate described above it should have been missing. Either the shard load silently failed — yielding size === 0 and a rebuild that then did not persist — or there is a re-index path we did not trace. Stating it rather than leaving it out, since it may indicate a third behaviour.

Environment

  • agentmemory 0.9.29, iii engine v0.11.2
  • Clean-room repro: node:22-bookworm-slim, stock npm package, fresh volume, default config
  • Originally observed on a 112 MB store (~99% graph) with LLM compression and local embeddings enabled, then reproduced with all of that removed

Related

#1180 (index saves scheduled only at boot) describes an adjacent symptom; this report identifies the teardown race and the size === 0 gate as the mechanism. #1179 (rebuild only runs on an empty index) is the same gate seen from the embeddings side.

Activity

  1. InfiniteSemicolon commented on Sep 12, 2026

    @InfiniteSemicolon

    I hit this on 0.9.29 with the v0.11.2 iii binary the CLI installs, outside Docker. It runs under systemd with ExecStop set to agentmemory stop, so the worker-first stop order from the 0.9.28 hotfix (#849) was in place. Three details from that run add to the report above.

    The engine's delay is a fixed 5 s persist tick. iii's file-based KV store acknowledges state::set once its in-memory map is updated. A single task writes the changed scopes to disk every save_interval_ms (default 5000), phase-locked to engine start, and nothing writes them on SIGTERM. The delay between a write and the disk therefore varies from 0 to 5 s by phase, and the ~2 s above is one sample from that range. I filed the engine side as iii-hq/iii#2177.

    The loss reaches the memory record. A POST /agentmemory/remember returned 201 at 19:38:08.368, systemctl stop began 4 ms later, and the stop finished at 19:38:09.338 with Result=success. While the service was stopped, mem:memories was unchanged, no new BM25 or vector shard generation existed, and the new id appeared in none of those files. That attempt lost every write since the previous tick, the new memory included. The three memories already on disk were intact. A boot-time index repair cannot bring the new memory back, because the store never persisted it.

    The worker-first order does not close it. runStop gives the worker up to 5 s to exit, SIGTERMs the engine the moment it does, and gives the engine up to 3 s before a SIGKILL. Both are upper bounds, so raising either one does not delay the engine's SIGTERM. The worker's indexPersistence.save() returned 9 ms after it started, and the whole stop took about one second, which leaves the engine a 1 s window inside a 5 s tick. A write at a random point in the tick is lost most of the time. In my five back-to-back attempts only the first lost its writes. The loop waited for the health check and then a fixed 8 s after each restart before writing, which put the last write of each shutdown save in attempts 2 to 5 less than 250 ms before a tick. Waiting at least one full tick after the last write would land what came before the wait, and it still leaves writes made during the wait unconfirmed.

    The CLI prints "Stopped. Memories persisted to disk" unconditionally, and it printed that line on the lossy attempt, as #843 also reported.

    For anyone who needs this closed before a release, here is what fixed it in my build. At the end of shutdown(), after the index save, the worker lists each scope it wrote and polls the matching <scope>.bin under the state store's file_path until the parsed content matches or an 8 s deadline passes. I also raised the worker grace in runStop from 5 s to 15 s. That change does nothing for the race itself; it keeps the CLI from SIGKILLing the worker during the wait. With no writes arriving during shutdown, a confirmed stop takes at most one tick plus about a second. The wait has the same kind of gap as a fixed delay: a write the still-registered worker accepts after its scope is confirmed, or to a scope first written during the wait, is not covered, and none of my attempts had a concurrent writer. Unregistering the worker's functions before the wait would close that. Every attempt kept the record: six at a phase where the unpatched build loses writes (five placed there on purpose) and four at a phase where it would have landed anyway. It depends on the engine's on-disk format, so a flush in iii is the better long-term fix.

    Two smaller notes on the rest of this thread. I applied #1180's first suggested shape, a debounced scheduleSave() at the vector-add site, and a live write reached the persisted vector shard within about 10 s with no restart. It sits behind the embedding-provider guard, so from the code it would not fire on an install with no provider; I did not test that. I also extended the existing legacy-gap loop that walks mem:memories at boot so it re-embeds memories missing from the vector index. That covers memories only. It leaves the idx.size === 0 gate and observations untouched, so it does not address the observation loss described here. On stock 0.9.29 that loop already backfills the BM25 entries of memories, so for those entries the boot repair is wider than the empty-index case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions