Repository navigation
Description
Current-main boundary
On main ab3e4efd282659b87d31b36c8514c6bc6b871f0b, newly generated vectors can all be lost if the first process lifetime ends before the default 600,000 ms checkpoint. Source observations survive and the keyword index rebuilds, so successful keyword retrieval can obscure the missing vector coverage. With no saved vector index, full backfill requires explicit opt-in.
This is narrower than the older shutdown and partial-index reports: current main already rebuilds the keyword index from stored content. Do not restore the old assumption that rebuild only occurs when an index is completely empty.
Reproduction and results
Use an isolated instance, a real 768-dimensional local embedding provider, 100 synthetic observations, and no chat compression:
- Capture all 100 observations through the built hooks. Confirm
vectorDocuments: 100. - Confirm
indexPersistence.saveIntervalMs: 600000,pendingChanges: 100, and no successful vector save yet. - Force-stop the worker and engine before the first checkpoint, then restart using the same data and default backfill policy.
- Observed: 100 source observations and 100 keyword documents remain, but
vectorDocuments: 0andpendingVectorBackfill: 100. - Ten exact-marker queries still pass through keyword retrieval. This is not proof that vector recovery succeeded.
Control: enable explicit full backfill and set AGENTMEMORY_INDEX_SAVE_INTERVAL_MS=1000. Wait for a completed checkpoint and an additional six-second persistence opportunity, then force-stop and restart with full backfill disabled. All 100 vectors survive. This confirms the existing checkpoint mechanism works when allowed to complete; one second is a test setting, not a proposed universal default.
Required behavior
- Define and document the first-checkpoint crash-loss window separately from steady-state batching.
- Evaluate an early bounded first checkpoint or equivalent persistence of completed vector work, using the existing storage mechanism.
- Keep checkpoint work bounded in CPU, memory and written bytes; do not require a full-index rewrite per observation.
- Report the difference between active recovery and recovery paused pending explicit approval. A nonzero pending count must not imply background work is running when full backfill is disabled.
- Preserve keyword fallback and the opt-in/budget boundary for full re-embedding. Do not recover by silently issuing unlimited provider calls.
Acceptance checks
Crash before the first checkpoint, after a completed checkpoint, and while later changes are pending. Assert vector counts and provider-call counts independently of keyword hits. Test absent and partially saved indexes. Verify status identifies paused versus running backfill and gives an accurate next action.
Source and existing work
Default interval, save scheduling, boot reconciliation, backfill status guidance.
Related earlier reports: #1180, #1335, #1179 and #1372. PRs #1177 and #1282 concern diagnostics and index maintenance. This follow-up is specifically the current first-checkpoint plus opt-in recovery boundary, not a duplicate of the old serialization or keyword-rebuild implementation.
Activity