Skip to content

Recover generated vectors after a crash before the first checkpoint #1439

Closed

Description

@rohitg00

Current-main boundary

On main ab3e4efd282659b87d31b36c8514c6bc6b871f0b, newly generated vectors can all be lost if the first process lifetime ends before the default 600,000 ms checkpoint. Source observations survive and the keyword index rebuilds, so successful keyword retrieval can obscure the missing vector coverage. With no saved vector index, full backfill requires explicit opt-in.

This is narrower than the older shutdown and partial-index reports: current main already rebuilds the keyword index from stored content. Do not restore the old assumption that rebuild only occurs when an index is completely empty.

Reproduction and results

Use an isolated instance, a real 768-dimensional local embedding provider, 100 synthetic observations, and no chat compression:

  1. Capture all 100 observations through the built hooks. Confirm vectorDocuments: 100.
  2. Confirm indexPersistence.saveIntervalMs: 600000, pendingChanges: 100, and no successful vector save yet.
  3. Force-stop the worker and engine before the first checkpoint, then restart using the same data and default backfill policy.
  4. Observed: 100 source observations and 100 keyword documents remain, but vectorDocuments: 0 and pendingVectorBackfill: 100.
  5. Ten exact-marker queries still pass through keyword retrieval. This is not proof that vector recovery succeeded.

Control: enable explicit full backfill and set AGENTMEMORY_INDEX_SAVE_INTERVAL_MS=1000. Wait for a completed checkpoint and an additional six-second persistence opportunity, then force-stop and restart with full backfill disabled. All 100 vectors survive. This confirms the existing checkpoint mechanism works when allowed to complete; one second is a test setting, not a proposed universal default.

Required behavior

  • Define and document the first-checkpoint crash-loss window separately from steady-state batching.
  • Evaluate an early bounded first checkpoint or equivalent persistence of completed vector work, using the existing storage mechanism.
  • Keep checkpoint work bounded in CPU, memory and written bytes; do not require a full-index rewrite per observation.
  • Report the difference between active recovery and recovery paused pending explicit approval. A nonzero pending count must not imply background work is running when full backfill is disabled.
  • Preserve keyword fallback and the opt-in/budget boundary for full re-embedding. Do not recover by silently issuing unlimited provider calls.

Acceptance checks

Crash before the first checkpoint, after a completed checkpoint, and while later changes are pending. Assert vector counts and provider-call counts independently of keyword hits. Test absent and partially saved indexes. Verify status identifies paused versus running backfill and gives an accurate next action.

Source and existing work

Default interval, save scheduling, boot reconciliation, backfill status guidance.

Related earlier reports: #1180, #1335, #1179 and #1372. PRs #1177 and #1282 concern diagnostics and index maintenance. This follow-up is specifically the current first-checkpoint plus opt-in recovery boundary, not a duplicate of the old serialization or keyword-rebuild implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions