Skip to content

feat(graph): support opt-in full provenance retention with bounded search indexes #1604

Open

Description

@masa6161

User problem

I need to revisit older decisions and trace the observations behind graph relationships in a long-running store. The source text can remain available while the graph loses its links to that source.

For example, an observation supports a node, then later observations add more source IDs to that node. A last-32 cap can remove the original link even though the observation still exists. Graph expansion from that observation can then miss the node. Keeping a backup or retrieving the observation by a known ID does not preserve this relationship in the active graph.

Could agentmemory support an opt-in mode that retains all accepted provenance links, while keeping ordinary search bounded? The existing default could stay unchanged. Complete provenance inspection could use paginated APIs rather than returning the entire history in every search.

Relationship to existing work

This follows up on a retention tradeoff already discussed upstream:

  • Issue #1168 addresses provenance growth. This comment identifies degraded reverse lookup from older observations after capping source IDs. This request focuses on that historical-traceability gap.
  • Issue #1450 proposes indexes to avoid whole-graph reads, but its observation index assumes each node retains only its last 32 sources.
  • PR #1454 caps canonical provenance, including import, mesh, and restore paths. This request would add a retention choice for deployments that need historical traceability.
  • PR #1559 provides bounded indexes and explicit migration that preserves oversized canonical fields. This request adds complete retention through subsequent writes and exhaustive source lookup to that foundation.

Source checked on October 23, 2026, at 1c99508. The canonical write path still applies MAX_GRAPH_SOURCE_OBSERVATIONS = 32. This is a feature request about that retention policy.

Evidence from an isolated copy

An earlier isolated Windows test used the unmodified standard compaction function at 857cf1b, native iii 0.22.1, and the file backend. The frozen input contained:

  • 140,501 nodes and edges combined.
  • 40,334,358 source references across those records.
  • 29,939 distinct source IDs.

After compaction, 14,548 of those source IDs no longer appeared in any node or edge's provenance. Node and edge identities remained unchanged. All corresponding observation or memory records still existed, and non-graph files were byte-identical.

This measures stored-link loss, not overall search accuracy. It is not a benchmark of a replacement index. No production data was changed, and no stored text was sent externally.

Required behavior

Full retention would cover existing canonical node, edge, and temporal-history links, plus links accepted afterward. Previously discarded links cannot be reconstructed by reindexing. Explicit deletion, forget, reset, and replacement would retain their documented semantics.

Correct per-record attribution must remain intact. Retention must not reintroduce attribution of an entire observation batch to every touched record.

Ordinary recall could remain top-k and budget-limited. Explicit provenance inspection would enumerate all retained links in both directions. Partial search projections must remain distinguishable from complete provenance.

Proposed minimum scope

  1. Apply full retention across normal updates, temporal updates, import, mesh, and restore. Separate attribute conflicts from source-link acceptance, including skip and timestamp conflicts. A limited projection must never overwrite canonical provenance.
  2. Reuse existing generation, migration, name, and adjacency infrastructure. Group provenance postings into size-bounded pages and shard index scopes where needed. Evaluate a reversible integer-ID dictionary separately. A new backend, custom codec, and canonical schema migration are not prerequisites.
  3. Add cursor-based traversal from a node or edge to all sources, and from a source to all nodes or edges. Cover history and supported memory sources. Preserve authorization and scope filters. Report incomplete indexing, exhausted budgets, and expired cursors explicitly.
  4. Validate on a frozen copy before switching indexes. Use bounded key enumeration and point reads, coordinating the engine dependency with Issue #1585. Keep pages consistent with their generation and revision. Verify their correspondence to canonical data after restart.

Removing the cap alone leaves journal, single-record, scope-size, and RAM limits unresolved. Exceeding supported limits must not silently discard provenance.

Acceptance criteria

  • Migration preserves canonical source IDs, counts, and existing order, with explicit duplicate semantics.
  • Complete forward and reverse traversal matches the original links, including old sources.
  • All write paths, retries, and conflict cases preserve the chosen retention contract.
  • Interrupted builds, concurrent updates, and restarts cannot publish a falsely complete or mixed-revision index. Rollback must not silently resume a truncating writer.
  • Copy-based tests measure index bytes, write cost, startup time, resident memory, retrieval quality, and historical traceability. A synthetic link-loss reproduction has not been run yet.

Would opt-in full retention fit the project's direction? If an existing issue or design covers these guarantees, I would like to follow it. Otherwise, could we agree on the retention and traversal contract before extending the index?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions