Skip to content

mem:audit is 71% index_persist rows the audit policy does not cover #1181

Closed

Description

@dmazhukov

mem:audit is dominated by rows the daemon writes about its own index flushes. On one store
it had reached 31 MB / 84 028 entries, of which 59 876 (71%) were index_persist. That
makes memory_audit take ~2.2 s and adds the same rows to every startup read.

Why these rows do not belong there

The policy at the top of src/functions/audit.ts (issue #125) defines what the log is for:

Every structural deletion of a memory, observation, session, or semantic row MUST call
recordAudit.

Index shard writes and manifest publishes delete no user rows, so they are outside that
definition. The same comment block also names the failure mode this runs into:

Per-item audit rows would flood the audit log during routine sweeps.

IndexPersistence.save() is a routine sweep on a 5 s debounce, and it emits one row per shard
write plus rows for the manifest publish and the legacy cleanup — three for a single-document
BM25 save, more as shard count grows and again when a vector index is present.

Reproduction

// with a KV double, or against a live daemon watching mem:audit
const persistence = new IndexPersistence(kv, bm25WithOneDoc, null);
await persistence.save();
// mem:audit now holds 3 index_persist rows for one flush

There is no retention, prune, or rotation anywhere to absorb this: no *AUDIT* env var, no
cap key, and recordAudit writes unconditionally.

Proposal

Make index_persist auditing opt-in behind AGENTMEMORY_AUDIT_INDEX_PERSIST, off by default,
so the log matches the policy its own module states. Index persistence keeps working
identically; only the audit rows stop.

Retention as a mechanism is deliberately out of scope here — it is a larger design question
and worth its own issue. This change is the part that needs no policy decision, because the
policy already exists and this code sits outside it.

A PR follows. Happy to reshape it if you would rather solve this with retention instead, or
keep the rows and batch them into one per flush.

Environment

  • agentmemory 0.9.29
  • Node v26.5.0
  • macOS 26.5.2, arm64

Related

#125 (the coverage policy this cites), #1115 (orphaned index generations — the same subsystem
growing unboundedly for a different reason), #766 (audit events for failures).

Activity

  1. Tomauskasz commented on Aug 13, 2026

    @Tomauskasz

    This change would remove the largest avoidable producer, but it would not bound KV.audit. Current queryAudit still loads and sorts the complete Audit scope before applying limit, and no retention or rotation operation exists.

    I opened #1192 for the separate retention scope that this Issue deliberately excludes: count/age/size reporting, dry-run pruning, export/checkpoint before deletion, configurable retention by operation class, and bounded queries.

    The two changes complement each other: this Issue lowers the growth rate; #1192 defines the storage boundary.

  2. dmazhukov commented on Aug 18, 2026

    @dmazhukov
    ContributorAuthor

    Confirmed on both counts. queryAudit (src/functions/audit.ts) loads the full KV.audit scope with kv.list, sorts it, then slices to limit, so the full set stays in memory regardless of the requested limit. #1182 only cuts the input rate for index_persist; it doesn't touch storage bounds. #1192 is the right home for retention: count/age/size accounting, dry-run pruning, and export-before-delete are a separate design question from this issue's opt-in gate.

  3. added a commit that references this issue on Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions