Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 37 additions & 217 deletions README.md

Large diffs are not rendered by default.

227 changes: 9 additions & 218 deletions READMEs/README.ar-SA.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.bn-BD.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.cs-CZ.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.da-DK.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.de-DE.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.el-GR.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.es-ES.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.fi-FI.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.fr-FR.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.he-IL.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.hi-IN.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.hu-HU.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.id-ID.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.it-IT.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.ja-JP.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.ko-KR.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.nb-NO.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.nl-NL.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.pl-PL.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.pt-BR.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.pt-PT.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.ro-RO.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.ru-RU.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.sv-SE.md

Large diffs are not rendered by default.

227 changes: 10 additions & 217 deletions READMEs/README.th-TH.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.tl-PH.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.tr-TR.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.uk-UA.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.ur-PK.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.vi-VN.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.zh-CN.md

Large diffs are not rendered by default.

229 changes: 10 additions & 219 deletions READMEs/README.zh-TW.md

Large diffs are not rendered by default.

190 changes: 18 additions & 172 deletions benchmark/COMPARISON.md
Original file line number Diff line number Diff line change
@@ -1,181 +1,27 @@
# AI Agent Memory: Benchmark Comparison
# Choosing a memory system

How agentmemory compares against other persistent memory solutions for AI coding agents.
Checked against the linked primary sources on 2026-10-28. Products evolve; deployment, model usage and commercial plans depend on the selected configuration.

All numbers here come from published benchmarks or public repositories. We link to primary sources wherever possible so you can reproduce.
| Project | Main use | Deployment and integration |
|---|---|---|
| **[AgentMemory](../README.md)** | Shared project memory for existing coding agents | Apache-2.0 local service; automatic capture on supported integrations, MCP and REST. BM25 works without a model key; embeddings and LLM processing are optional. The CLI installs its runtime. |
| **[Mem0](https://docs.mem0.ai/introduction)** | Memory APIs for applications and coding agents | Hosted platform or open-source library/server; SDKs, MCP and coding-agent plugins. Capture depends on the integration. [Platform and OSS differ](https://docs.mem0.ai/platform/platform-vs-oss), including graph availability. |
| **[Supermemory](https://supermemory.ai/docs/self-hosting/overview)** | Memory and document context for applications and agents | Hosted API or local server binary, plus coding-agent plugins. The local server has embedded storage and local embeddings; its codebase is separate and non-public. Local and hosted capabilities differ. |
| **[Zep / Graphiti](https://github.com/getzep/graphiti)** | Enterprise context and temporal knowledge graphs | Zep offers a managed or BYOC platform. Apache-2.0 Graphiti is a separate self-hosted graph framework with database and model configuration. |
| **[Letta](https://docs.letta.com/)** | Stateful agents with managed memory | Agent harness, CLI and SDK with MemFS/shared memory; managed cloud or self-hosted runtime. It supplies the agent runtime as well as memory. |
| **[Cognee](https://github.com/topoteretes/cognee)** | Searchable knowledge from documents, code and conversations | Open-source local library/server or Cognee Cloud, with plugins and MCP. Local extraction and embedding models support a keyless path. |
| **[claude-mem](https://github.com/thedotmack/claude-mem)** | Capture and summarize coding sessions | Apache-2.0 project with multiple agent integrations. Model-backed summaries can use its hosted observer or a configured provider; installation and provider choices affect data flow and cost. |

---
AgentMemory fits developers who want one local memory service shared by their existing tools, with inspectable observations, lessons, actions, source expansion and a live viewer. An application-facing API, managed enterprise platform, document graph or complete stateful-agent runtime may suit a different workload better.

## Retrieval Accuracy (LongMemEval)
## Compare measurements on the same terms

[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) measures long-term memory retrieval across ~48 sessions per question on the S variant (500 questions, ~115K tokens each).
The [current retrieval report](../docs/benchmarks/2026-10-28-retrieval-evaluation.md) measures the BM25 component and a grep baseline over identical supplied sessions. It includes strict recall-all@5, partial recall, any-hit, coverage and failures. It does not rank the products above.

| System | Benchmark | R@5 | Notes |
|---|---|---|---|
| **agentmemory** (BM25 + Vector) | LongMemEval-S | **95.2%** | `all-MiniLM-L6-v2` embeddings, no API key |
| agentmemory (BM25-only) | LongMemEval-S | 86.2% | Fallback when no embedding provider available |
| MemPalace | LongMemEval-S | ~96.6% (self-reported) | Vendor-published number we have not independently reproduced. Vector-only with a larger embedding model and no agent-integration surface (no hooks, no MCP, no multi-agent) |
| oracleagentmemory | LongMemEval | 94.4% (self-reported) | Vendor-published, scored with GPT-5.5 at "xhigh reasoning" and requires an Oracle AI Database. We have not reproduced it. agentmemory's 95.2% uses free local embeddings and no API key |
| Letta / MemGPT | LoCoMo | 83.2% | Different benchmark (LoCoMo, not LongMemEval) |
| Mem0 | LoCoMo | 68.5% | Different benchmark (LoCoMo, not LongMemEval) |
A useful comparison must hold the dataset revision, ingestion path, retrieved unit, cutoff, model, context budget, reader and judge constant. Retrieval recall, answer accuracy, task completion and token use measure different things. Historical scores and unrelated datasets should not share a leaderboard.

**⚠️ Apples vs oranges caveat:** only agentmemory's 95.2% is our own measured result, reproducible from the methodology below. Every other number here is the vendor's published claim, on a different benchmark or harness, that we have not independently reproduced: MemPalace and oracleagentmemory report LongMemEval (oracleagentmemory's run used GPT-5.5 at "xhigh reasoning" against an Oracle AI Database), while Letta and Mem0 publish on [LoCoMo](https://snap-stanford.github.io/LoCoMo/). Treat them as ballpark vendor claims, not a head-to-head on identical data. We'd love to run every system on the same dataset; if any maintainer wants to collaborate, open an issue.
## Costs and data flow

Full agentmemory methodology: [`LONGMEMEVAL.md`](LONGMEMEVAL.md)
AgentMemory has no license fee or hosted subscription. You operate its local runtime; optional hosted models charge for their usage. Installation, updates and first-use local model downloads need network access. Memory text stays local with keyless search or local models; configured remote services receive the text they process.

---

## Feature Matrix

| Feature | agentmemory | mem0 | Letta/MemGPT | Khoj | supermemory | MemPalace | oracleagentmemory | Hippo |
|---|---|---|---|---|---|---|---|---|
| **GitHub stars** | Growing | 58K+ | 23K+ | 35K+ | 26K+ | 54K+ | PyPI (Oracle) | Trending |
| **Type** | Memory engine + MCP server | Memory layer API | Full agent runtime | Personal AI | Memory API + app | Benchmark-focused OSS | Memory engine (Oracle DB) | Memory system |
| **Auto-capture via hooks** | ✅ 12 lifecycle hooks | ❌ Manual `add()` | ❌ Agent self-edits | ❌ Manual | ❌ API-side extraction | ❌ Manual | ❌ API extraction | ❌ Manual |
| **Search strategy** | BM25 + Vector + Graph | Vector + Graph | Vector (archival) | Semantic | Vector + RAG | Vector-only (large model) | Vector + semantic | Decay-weighted |
| **Multi-agent coordination** | ✅ Leases + signals + mesh | ❌ | Runtime-internal only | ❌ | ❌ | ❌ | Scoped only (user/agent/thread) | Multi-agent shared |
| **Framework lock-in** | None | None | High | Standalone | None (drop-in wrappers) | None | Oracle Database | None |
| **External deps** | None | Qdrant/pgvector | Postgres + vector | Multiple | Managed cloud | Vector store | Oracle AI Database | None |
| **Self-hostable** | ✅ default | Optional | Optional | ✅ | ❌ Cloud-only | ✅ | ✅ (needs Oracle DB) | ✅ |
| **Knowledge graph** | ✅ Entity extraction + BFS | ✅ Mem0g variant | ❌ | Doc links | ❌ | ❌ | ❌ | ❌ |
| **Memory decay** | ✅ Ebbinghaus + tiered | ❌ | ❌ | ❌ | ✅ Auto-forget | ❌ | ❌ | ✅ Half-lives |
| **4-tier consolidation** | ✅ Working → episodic → semantic → procedural | ❌ | OS-inspired tiers | ❌ | ❌ | ❌ | ❌ | Episodic + semantic |
| **Version / supersession** | ✅ Jaccard-based | Passive | ❌ | ❌ | ✅ Auto-resolve | ❌ | ❌ | ❌ |
| **Real-time viewer** | ✅ Port 3113 | Cloud dashboard | Cloud dashboard | Web UI | Cloud dashboard | ❌ | ❌ | ❌ |
| **Privacy filtering** | ✅ Strips secrets pre-store | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Obsidian export** | ✅ Built-in | ❌ | ❌ | Native format | ❌ | ❌ | ❌ | ❌ |
| **Cross-agent** | ✅ MCP + REST | API calls | Within runtime | Standalone | MCP + API | Standalone | Python API | Multi-agent shared |
| **Audit trail** | ✅ All mutations logged | ❌ | Limited | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Language SDKs** | Any (REST + MCP) | Python + TS | Python only | API | Python + TS | Python | Python only | Node |

---

## Token Efficiency

The main reason to use persistent memory at all: token cost. Here's what one year of heavy agent use looks like across approaches.

| Approach | Tokens / year | Cost / year | Notes |
|---|---|---|---|
| Paste full history into context | 19.5M+ | Impossible | Exceeds context window after ~200 observations |
| LLM-summarized memory (extraction-based) | ~650K | ~$500 | Lossy — summarization drops detail |
| **agentmemory (API embeddings)** | **~170K** | **~$10** | Token-budgeted, only relevant memories injected |
| **agentmemory (local embeddings)** | **~170K** | **$0** | `all-MiniLM-L6-v2` runs in-process |
| supermemory | Not published | Cloud pricing | Managed API, no local token budget |
| Mem0 | Varies by integration | Varies | Extraction-based, no token budget |

**agentmemory ships with a built-in token savings calculator.** Run `npx @agentmemory/agentmemory status` after a few sessions and you'll see exactly how many tokens you've saved vs. pasting the full history.

---

## What Each Tool Is Best At

This isn't a "agentmemory wins everything" page. Different tools solve different problems.

**Choose agentmemory if you want:**
- Automatic capture with zero manual `add()` calls
- MCP server that works across Claude Code, Cursor, Codex, Gemini CLI, etc.
- Hybrid BM25 + vector + graph search
- Real-time viewer to see what your agent is learning
- Self-hostable with zero external databases
- Privacy filtering on API keys and secrets
- Multi-agent coordination (leases, signals, routines)

**Choose Mem0 if you want:**
- Framework-agnostic API to bolt onto an existing agent
- Managed cloud option with a dashboard
- Python + TypeScript SDKs for direct integration
- Entity/relationship extraction as the primary abstraction

**Choose Letta/MemGPT if you want:**
- A full agent runtime, not just memory
- OS-inspired memory tiers (core/archival/recall)
- Agents that self-edit their memory via function calls
- Long-running conversational agents (weeks/months)

**Choose Khoj if you want:**
- A personal AI second brain, not agent infrastructure
- Document-first search over your files and the web
- Obsidian/Notion/Emacs integrations
- Scheduled automations and research tasks

**Choose supermemory if you want:**
- A managed memory API with server-side auto-extraction and automatic forgetting
- Drop-in wrappers for major AI frameworks (Vercel AI, LangChain, LangGraph)
- A hosted dashboard with no infrastructure to run yourself
- RAG plus memory served from a single query

**Choose MemPalace if you want:**
- A simple, free, open-source vector memory store
- To chase its self-reported retrieval benchmark (we have not reproduced it)
- Pure retrieval over agent workflow features
- Note: no auto-capture, no MCP, no multi-agent coordination, so you wire all integration yourself

**Choose oracleagentmemory if you want:**
- You already run on Oracle AI Database and want memory inside it
- Enterprise Oracle stack with vector search in the same database
- LLM-backed extraction and are fine paying for a frontier model (their benchmark used GPT-5.5)
- Note: Python-only, Oracle Database required, no MCP, no real-time viewer

**Choose Hippo if you want:**
- Biologically-inspired memory model (decay, consolidation, sleep)
- Multi-agent shared memory as a primary feature
- "Forget by default, earn persistence through use" philosophy

**Choose TencentDB Agent Memory if you want:**
- Team-level shared memory: conversations, docs, and code turned into four asset types (Chat Memory, Skill, Wiki, CodeGraph) with team roles and ownership
- Zero-integration capture via an LLM proxy (point the agent's base URL at it; no hooks or MCP required)
- CodeGraph impact analysis (symbols, call relationships) alongside memory
- Note: the proxy sits in front of every model call, deployment is a multi-service Docker stack (Core + Hub + Proxy), the published benchmark is PersonaMem (76%, self-reported), and automated memory routing is still in progress per their README

**Choose Zep / Graphiti if you want:**
- A temporal knowledge graph: facts carry a time dimension, so "what was true when" is a first-class query
- The strongest published temporal-query results (LongMemEval 63.8%)
- Note: graph construction runs in the background, so freshly ingested facts can take time to become retrievable, and per-conversation memory footprint is reported to run far above extraction-based systems

**Choose Cognee if you want:**
- Knowledge-graph construction from documents and structured data before query time
- Entity-relationship extraction as the primary product rather than session capture
- Note: Python-only, and built for document ingestion rather than coding-agent memory

---

## Running Your Own Benchmarks

We encourage you to measure this yourself rather than trust any README. Here's how:

```bash
# Clone the repo
git clone https://github.com/rohitg00/agentmemory.git
cd agentmemory && npm install

# Run LongMemEval-S
npm run bench:longmemeval

# Run quality benchmark (240 observations, 20 queries)
npm run bench:quality

# Run scale benchmark
npm run bench:scale

# Run real embeddings benchmark
npm run bench:real-embeddings
```

Results land in `benchmark/results/`. All scripts, datasets, and results are committed for reproducibility.

---

## Corrections Welcome

If you maintain one of these tools and we got a number wrong, please open an issue or PR. We'd rather have accurate numbers than convenient ones.

If you want to add your tool to this comparison, open a PR with:
1. A link to your benchmark methodology
2. The metric and dataset you're measuring on
3. A commit hash / version so we can reproduce

**Sources:**
- Mem0 LoCoMo benchmark: [mem0.ai blog](https://mem0.ai)
- Letta LoCoMo benchmark: [letta.com/blog/benchmarking-ai-agent-memory](https://letta.com/blog/benchmarking-ai-agent-memory)
- LongMemEval paper: [arxiv.org/abs/2410.10813](https://arxiv.org/abs/2410.10813)
- LoCoMo paper: [snap-stanford.github.io/LoCoMo](https://snap-stanford.github.io/LoCoMo/)
For commercial services, use their current plans instead of a copied monthly price: [Mem0](https://mem0.ai/pricing), [Supermemory](https://supermemory.ai/pricing), [Zep](https://www.getzep.com/pricing/), [Letta](https://docs.letta.com/), [Cognee](https://docs.cognee.ai/cognee-cloud/overview), and [claude-mem](https://github.com/thedotmack/claude-mem). Check the chosen deployment's provider, telemetry, retention and access controls before using private data.
4 changes: 3 additions & 1 deletion benchmark/LONGMEMEVAL.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# LongMemEval-S Benchmark Results
# Historical LongMemEval-S Benchmark Results

This report uses historical **any-hit@k** scoring and an older input pipeline. It is not a 0.9.31 strict-recall result. See the [current evaluation and receipts](../docs/benchmarks/2026-10-28-retrieval-evaluation.md); the metrics and denominators differ.

[LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is an academic benchmark for evaluating long-term memory in chat assistants. It tests 5 core abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.

Expand Down
Loading
Loading