Skip to content

Scheduled Git snapshots grow without a retention policy #1158

Open

Description

@Tomauskasz

User problem

With scheduled Git snapshots enabled, the snapshot repository grows continuously because every changed state.json is committed and no age-, count-, or size-based retention policy exists.

I cannot leave hourly snapshots enabled indefinitely without separately maintaining the Git repository or accepting unbounded disk growth.

Environment

  • agentmemory: 0.9.28
  • upstream main checked at: d60652a7058773fa9428fa720eda38942f12f014
  • Node: v24.15.0
  • OS: macOS 26.6 arm64
  • snapshot interval: 3600 seconds

Reproduction

  1. Set SNAPSHOT_ENABLED=true and SNAPSHOT_INTERVAL=3600.
  2. Let the canonical store change between snapshot runs.
  3. Allow scheduled and manual snapshots to accumulate.
  4. Inspect the snapshot directory with du -sh, git count-objects -vH, and git rev-list --count HEAD.

One current installation with 48 snapshot commits has:

  • 327 MiB snapshot directory;
  • 151 MiB current state.json;
  • 161 MiB .git directory;
  • 131.84 MiB of loose Git objects plus a 29.45 MiB pack.

The commit count includes both scheduled and supervised/manual snapshots. The important behavior is that all reachable history remains indefinitely and AgentMemory exposes no retention control.

Actual result

mem::snapshot-create writes the complete state export and runs git commit. It never expires commits, compacts scheduled history, or reports a retention boundary:

mem::snapshot-list displays only 20 commits, but older commits remain reachable in Git:

Desired behavior

Provide an explicit, documented retention policy for scheduled snapshots. The policy should bound history by age, count, or size while preserving at least the latest verified snapshot and any snapshots that the operator deliberately protects.

The exact policy can remain configurable. The required user outcome is that enabling hourly snapshots does not imply unbounded storage growth.

A safe implementation should also report what it retained or pruned and compact unreachable Git objects after pruning. Retention must fail closed if snapshot verification or repository operations fail.

Related work

Activity

  1. Tomauskasz commented on Aug 13, 2026

    @Tomauskasz
    Author

    Confirmed against current main at 2973e4e: merged PR #1136 now runs the Snapshot timer, while Snapshot history still has no retention boundary described here.

    There is a separate recovery-contract problem: mem::snapshot-create exports only five data groups, and mem::snapshot-restore performs additive kv.set operations without clearing current rows or rebuilding search and Graph indexes. I opened #1190 for Snapshot completeness and exact restore semantics.

    These issues are independent but interact operationally: retention must preserve the latest verified complete Snapshot. Pruning history before Snapshot verification would leave an operator with bounded storage but no reliable recovery point.

  2. aberenyi commented on Nov 1, 2026

    @aberenyi

    Another data point, from agentmemory 0.9.31 on macOS arm64 with Apple Git 2.54.0 and SNAPSHOT_INTERVAL=3600.

    After 1,134 snapshot commits since 2026-09-08:

    • state.json is 368 MB.
    • .git is 21 GB: 18.2 GiB of loose objects (627) and 568 MiB in 3 packs.

    The growth also caused a CPU and memory problem. Each git commit in mem::snapshot-create starts a detached git maintenance run --auto, which is reparented to PID 1. This repo had a local gc.auto=64 (not set by agentmemory), so every snapshot started a git repack -d -l --geometric=2 --write-midx. Each repack took longer than the one-hour snapshot gap, and nothing stopped a new one from starting while the old one still ran. I found six running at the same time. The oldest had run for 6.5 hours. Each used 55–128% CPU, together they used about 8.5 GB of RAM, and they had left 1.9 GiB of tmp_pack_* files.

    So a retention policy alone may not be enough. After pruning, agentmemory should run one controlled compaction itself and stop git's automatic maintenance on every snapshot commit, for example with git -c maintenance.auto=false commit.

    Workaround used here: kill the repacks, delete .git/objects/pack/tmp_pack_*, and set gc.auto=0 and maintenance.auto=false in the snapshot repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions